UP | HOME

What are Large Language Models (LLMs)

Large, general-purpose language models can be pre-trained and then fine-tuned for specific purposes.

Source: Introduction to Large Language Models by Goodle.

Marqo.ai

What is tensor search?

Tensor search uses deep-learning to transform documents, images and other data into collections of vectors called “tensors”. Representing data as tensors allows us to match queries against documents with human-like understanding of the query and document’s content. Tensor search can power a variety of use cases such as:

  • end user search and recommendations
  • multi-modal search (image-to-image, text-to-image, image-to-text)
  • chat bots and question and answer systems
  • text and image classification

LLM-150

DONE Create an s3 bucket

Create a lambda that triggers on s3 bucket creation

DONE Create Lambda with Python

DONE Receive document from s3 bucket

DONE Get chunks of document

INPROGRESS integrate marqo

INPROGRESS upload document chunks to marqo

For the indexing I thought I can get the folder path of the file where we can have somo sort of hierarchy like:

  • businnes_unit/group-id[user-id]/*

    translated that would be something like:

  • stat/organization[user or domain or group?])/file.pdf

    So I need to create the name of the index taking into account those params.

    • bussiness unit
    • group

Constraints:

  • filename must not have ’+’ chars, because if the file has blanks, S3 changes it to + chars.

Josue1:9BeBrave