feat(llm): Add Ollama provider and PyMuPDF image extraction

This commit introduces support for Ollama as an alternative Large Language Model (LLM) provider and enhances PDF image extraction capabilities.

- **Ollama Integration:**
    - Implemented `set_ollama_config` to configure Ollama's base URL from `config.ini`.
    - Modified `llm.py` to dynamically select and configure the LLM (Gemini or Ollama) based on the `PROVIDER` setting.
    - Updated `get_model_name` to return provider-specific default model names.
    - `pdf_convertor.py` now conditionally initializes `ChatGoogleGenerativeAI` or `ChatOllama` based on the configured provider.
- **PyMuPDF Image Extraction:**
    - Added a new `extract_images_from_pdf` function using PyMuPDF (`fitz`) for direct image extraction from PDF files.
    - Introduced `get_extract_images_from_pdf_flag` to control this feature via `config.ini`.
    - `convert_pdf_to_markdown` and `refine_content` functions were updated to utilize this new image extraction method when enabled.
- **Refinement Flow:**
    - Adjusted the order of `save_md_images` in `main.py` and added an option to save the refined markdown with a specific filename (`index_refined.md`).
- **Dependencies:**
    - Updated `pyproject.lock` to include new dependencies for Ollama integration (`langchain-ollama`) and PyMuPDF (`PyMuPDF`), along with platform-specific markers for NVIDIA dependencies.
This commit is contained in:
2025-11-11 22:35:23 +11:00
parent 2c6c2c1078
commit 26951b8bc0
7 changed files with 773 additions and 72 deletions

View File

@@ -29,12 +29,14 @@ def main():
with open(pdf_path, "rb") as pdf_file:
pdf_content = pdf_file.read()
md, images = convert_pdf_to_markdown(pdf_content)
save_md_images(current_output_dir, md, images)
try:
md = refine_content(md, images, pdf_content)
except BaseException:
continue
save_md_images(current_output_dir, md, images)
save_md_images(current_output_dir, md, images, md_name="index_refined.md")
if __name__ == "__main__":