Guide to creating Lunes

First, is creating and using Lunes free?

For now yes! Sign in with Google and try exploring available Lunes!

How does training a Lune work?

There are two ways to add knowledge sources to a Lune.

During initial creation
Adding additional sources after creation

During creation, you can add as many URLs as you like (repecting the limitations below). Post-creation, you can add a single URL each time.

For each URL, you can specify Full or Single for the ingestion type.

Full will crawl all child links on the input page. Single will only generate dynamic embeddings on the content of the current page.

It is very important to understand that the URL you are inputting (if you specified Full), will be treated as the parent URL. For example, if you want to get all of the docs for developer.example.com and they serve their documentation at developer.example.com/docs, then developer.example.com/docs/authentication/oauth would not get all of the documentation under /docs.

What are the current limitations?

At the moment, Lune is in its infant stage. While many others in our shoes might choose to conduct a closed beta, we believe strongly in being transparent building Lune and our current limitations by opening up the platform to the public.

For the foreseeable future, many thing break. There will be outages. Sometimes a Lunes might give you crap answers. But we are working hard around the clock to make sure we're improving the user experience every single day.

Here are some limitations you will face right now:

You can only create up to four Lunes at a time. If you delete one, you may train another.
Training times will vary. Most Lunes will finish under 10 minutes but some may go over, we hope to bring that time down further over the next few days.
You can't can't add single knowledge source that it too large (>200 child URLs, more on that below).
You currently can't add sources with dynamically loaded content (you could, but the resulting embeddings created would probably be useless)
We do not support local files as of now, only content served through the web and rendered out in html (no images/graph support as of now)

More on knowledge source limitations

The most important constraint to be aware of right now are sources with dynamically loaded content. Any Lunes trained on such sources will not perform any better than GPT-4o(the underlying model powering Lunes) on that given topic.

Until we implement better ways to handle that, if you aren't sure, you can use this command to download the page and check if the content or child links you wish to train the Lune on are present.

curl -o filename.html https://example.com

The other major limitation (for now) is that a parent URL can't have more than 200 total child links, including all nested links down the entire tree. This limitation is only relevant if you set the ingestion type to Full for a given input source. If a given single knowledge source exceeds this limit, then the training will be cut off for that session at 200 URLs.

A recommend work around is to manually input each URL for every page you want included and set the ingestion type to Single or break up the larger parent page into smaller training sessions, by adding additional knowledge sources one at a time. We understand this may be inconvenient, and working to scale our backend infrastructure to handle larger training requests.

Most small-medium sized documentation or references will be under this limit. But here are some examples to help clarify.

Under the limit:

Over the limit for single training session:

https://developer.x.com/en/docs/x-api (and it has dyamically fetched content)
https://www.tensorflow.org/api_docs (for something like this, start by training a Lune on a subset of the documentation, like just for TF Python)
https://cloud.google.com/docs (way to big, hopefully this is something we can tackle as a community supported public Lune in the near future)

Tips for creating Lunes

Even though we don't support files directly right now, for things like READMEs hosted on github, you can click into the file and view the raw output like below which works well:

https://raw.githubusercontent.com/sickcodes/Docker-OSX/master/README.md (we already have a Lune on this by the way)

Try not to include sources that rely heavily on images, as Lune will not be able to properly ingest those sources. Generaly, Lune will work better with sources that are clean and organized.

Finally, make sure you first check the Explore section to view Lunes and see if a similar public Lune already exists.

Questions? Reach out to us at support@trylune.ai and we'll try to be in touch within 24 hours.