Indexes

This document outlines the process for scraping text from internet sources and document files, splitting text, and inserting documents into a vector database using the Prompt Engineers platform.

Step 1: Scraping Text

1. Select Source: Choose 'Files' or other sources like 'Text', 'YouTube', 'URLs', etc., from the 'Select Loader' dropdown.

2. Upload Files: Click 'Choose Files' to upload documents (supports .txt, .pdf, .csv).

3. View Document: If you're uploading a PDF, it can be previewed in the interface.

Step 2: Text Splitting

1. Select Splitter: From the 'Select Splitter' dropdown, choose a method for splitting text (e.g., 'Spacy', 'Python', 'Markdown').

2. Set Overlap: Configure 'Chunk Overlap' to determine how much overlap each chunk of text should have (useful for ensuring context in analysis).

Step 3: Upserting Documents

1. Collect Text: Click 'Collect' to start the scraping process. The system may display progress, such as "Splitting page 12 into chunks."

2. Review Chunks: Once the document is processed, review the extracted text chunks.

3. Upsert Documents:

   • New Index: Select 'New', enter an 'Index Name', and click 'Create'.

   • Existing Index: Select 'Existing', choose the index, and upsert the text.

Notes

• Ensure the proper selection of the splitter and loader to match the document type and desired chunking strategy.

• The 'Upsert' process allows for adding new documents or updating existing ones in the vector database.

• Monitor the progress bar or messages to ensure that the scraping and upserting are proceeding correctly.

---

This workflow enables users to efficiently process and store large amounts of text data, preparing it for advanced search and retrieval in vector databases.