[🇧🇷 Português] [🇬🇧 English]

Investor Intelligence Platform 🇧🇷 Brazilian FIIs

Real Estate Investment Funds (FIIs) - Market Intelligence & Behavioral Analytics

$$\Huge {\textbf{\color{green} CRISP-DM} \space \textbf{\color{white} •} \space \textbf{\color{yellow} Data Lakehouse} \space \textbf{\color{white} •} \space \textbf{\color{green} NLP} \space \textbf{\color{white} •} \space \textbf{\color{yellow} Responsible AI} \space \textbf{\color{white} •} \space \textbf{\color{green} Regulatory Alignment}}$$

An institutional-grade intelligence platform for monitoring, structuring, ranking, and interpreting Brazilian Real Estate Investment Fund (FII) signals from financial media, research portals, and investor communities.

Big Data • PySpark • MapReduce Word Count • NLP • TF-IDF • BM25 Ranking • Hybrid Retrieval • FAISS + Multilingual Embeddings • Web Scraping • TOFU/MOFU/BOFU • CRISP-DM • FastAPI • Streamlit • Docker • Responsible AI • LGPD • EU AI Act Alignment

🎓 Academic

Institution: Pontifical Catholic University of São Paulo (PUC-SP)
School: FACEI — Faculty of Exact Sciences and Informatics
Bachelor’s Program: Human-Centerd AI & Data Science • 5th Semester • 2026
Course: AI Security, Cybersecurity & Social Engineering
Methodology: CRISP-DM (Cross-Industry Standard Process for Data Mining)
Professors ✨ Carlos Eduardo Paes and ✨ Eduardo Savino Gomes
Project Author: Fabiana ⚡️ Campanari

Tip

↗ Explore the Full Course Repository)

Real Estate Investment Funds (FIIs) 🇧🇷 Market Intelligence & Behavioral Analytics

A scalable platform combining Big Data, PySpark, MapReduce, Word Count, NLP, TF-IDF, BM25, FAISS, pMultilingual Embeddings, Web Scraping, Hybrid RAG and AI-Assisted Analytics to transform large-scale financial discussions into actionable insights for FIIs.

$$\Huge {\textbf{\color{green} Where market discussions become investment narratives…}}$$

$$\Huge {\textbf{\color{yellow} because markets talk a lot...}}$$

$$\Huge {\textbf{\color{green} intelligent systems just listen better}}$$

⚡

Warning

⚠️ Projects may be publicly shared when permitted. The focus is on applied, hands-on learning with real datasets in AI governance and security contexts. All sensitive content remains protected in private repositories when required.

⚠️ Disclaimer Plataforma exclusivamente educacional e analítica. Não constitui recomendação de investimento.

Table of Contents

1. Academic Context
2. Overview
3. System Overview
4. What This Platform Delivers
5. Why This Matters
6. Architecture and Pipeline
7. Notebooks Executive Summary
8. Notebooks NB00–NB07: Technical Report
9. The 3 Core Techniques + FAISS Semantic Layer
10. Data Sources — 21 Sources
11. Big Data Infrastructure
12. Dependencies and requirements.txt
13. CRISP-DM Methodology
14. Marketing Funnel: TOFU, MOFU and BOFU in the Project
15. RAG Chatbot: Groq + Gemini (Automatic Fallback)
16. Governance
17. Folder Structure
18. How to Run
19. Deployment & Automation Workflow
20. Which requirements*.txt to Use in Each Scenario
21. Makefile — Full Reference
22. Expected Outputs
23. Technical Glossary
24. Repository and Project Links
25. References

🎓 Academic and Institutional Context

This project was developed at PUC-SP in the courses of cybersecurity, social engineering, data engineering and Big Data analytics applied to financial markets. The original requirement focused on demonstrating a distributed word count solution using PySpark and the MapReduce paradigm.

From this starting point, the repository was extended to incorporate more advanced analytical techniques (TF-IDF, BM25, contextual sentiment), a serving architecture with FastAPI + RAG and a structured pipeline oriented towards FII marketing intelligence.

Academic Requirements Met

Requirement	Implementation
Distributed Computing	PySpark · RDD MapReduce · SparkSession
Big Data Architecture	Medallion (Bronze → Silver → Gold)
Machine Learning	TF-IDF · BM25 · Semantic Embeddings · Sentiment Analysis
Vector Search	FAISS · Dense Index · Multilingual PT-BR Embeddings
NLP	PT-BR Tokenization · FII Lexicon · Signal Flags
Data Governance	LGPD · EU AI Act · Responsible AI · XAI
REST API	FastAPI · Uvicorn
RAG / LLM	Groq (openai/gpt-oss-20b, primary) + Gemini 2.5 Flash (automatic fallback)
Visualization	Streamlit · Plotly
Cybersecurity	Narrative surface analysis · Social Engineering awareness

Product Overview and Definition

The Investor Intelligence Platform 🇧🇷 FIIs Brazil is not just an academic Big Data exercise. It is an investor intelligence platform for Brazilian Real Estate Investment Funds (FIIs), designed to transform fragmented public financial discussions into structured, searchable, explainable and decision-oriented market intelligence.

Instead of being a simple dashboard, the system operates as an end-to-end analytical environment that:

- collects data from 21 sources (RSS · scraping · Reddit)
-organizes them in a Bronze/Silver/Gold architecture
- enriches them with hybrid retrieval (TF-IDF + BM25 + FAISS semantic search with multilingual PT-BR embeddings), FII PT-BR sentiment and explainable marketing intelligence signals
- exposes results via FastAPI + RAG + Groq chatbot + Streamlit

%%{init:{
'theme':'dark',
'themeVariables':{
'background':'#090d13',
'primaryTextColor':'#F5F7FA',
'lineColor':'#2dd4bf'
}}}%%

graph LR

SRC["21 SOURCES<br/>RSS • Scraping • Social"]:::setup

PIPE["NLP PIPELINE<br/>PySpark • MapReduce<br/>TF-IDF • BM25 • FAISS"]:::gold

GOLD["GOLD LAYER<br/>Parquet artifacts"]:::bronze

API["FASTAPI<br/>REST + RAG"]:::dash

DASH["STREAMLIT<br/>Analytics Dashboard"]:::dash

LLM["LLM LAYER<br/>Groq + Gemini"]:::llm

SRC --> PIPE --> GOLD
GOLD --> API
GOLD --> DASH
API --> LLM
DASH --> LLM

classDef setup fill:#0d2137,stroke:#00d2ff,color:#F5F7FA,stroke-width:2.5px;
classDef bronze fill:#2a1512,stroke:#a85a4a,color:#F5F7FA,stroke-width:2.5px;
classDef silver fill:#1b2430,stroke:#b0b7c3,color:#F5F7FA,stroke-width:2.5px;
classDef gold fill:#2a2208,stroke:#e6c35a,color:#F5F7FA,stroke-width:2.5px;
classDef dash fill:#06363d,stroke:#2dd4bf,color:#F5F7FA,stroke-width:2.5px;
classDef llm fill:#231433,stroke:#b56cff,color:#F5F7FA,stroke-width:2.5px;

Loading

🎯 Objectives

1. Complete end-to-end pipeline — from ingestion to analytical output.
   
2. Distributed processing + NLP — PySpark MapReduce combined with TF-IDF, BM25 and contextual sentiment.
   
3. RAG over FII corpus — hybrid retrieval via TF-IDF, BM25 and FAISS-backed multilingual PT-BR embeddings, followed by contextual generation via Groq.
   
4. Cybersecurity and Social Engineering — security perspective in interpreting channels and narratives.

👥 Target Audience

- Asset and fund managers who monitor investor perception
- Financial analysts who track market narratives
- Marketing teams interested in FII visibility and engagement
- Academic evaluators assessing Big Data, Spark, NLP and RAG
- Recruiters and technical portfolio reviewers

Why This Matters ❓

Analysts, managers and financial communication teams face:

- information dispersed across dozens of portals and communities
- high noise-to-signal ratio in market discussions
- difficulty tracking how sentiment and narratives evolvev
- lack of transparent tools aligned with LGPD and the EU AI Act

Tip

This platform addresses this gap with 21 monitored sources, a Bronze/Silver/Gold pipeline and reproducible, interpretable analytics.

Source Coverage

The platform monitors a curated set of editorial and behavioral sources relevant to the Brazilian FII ecosystem. Instead of treating all inputs as an undifferentiated corpus, the project distinguishes:

- Editorial RSS sources — collected via structured feeds >
- Editorial portals via scraping — controlled extraction of public metadata >
- Behavioral social sources — Reddit as a community sentiment layer

Important

Detailed documentation per source: docs/data_sources.md

Collection Strategy by Source Type

RSS-First

When available, RSS is preferred: lower extraction cost, native structured metadata, no risk of breakage due to HTML layout changes, reliable scheduling.

Scraping as Controlled Fallback

When RSS is unavailable or unstable, controlled HTML extraction of public pages. It does not simulate human navigation — it collects observable metadata (titles, links, timestamps, categories, excerpts).

Collection of Social Sources

Reddit follows a separate logical path because it represents conversational and community-based data. It is treated as a behavioral and discursive input layer that complements editorial coverage with public sentiment and emerging narratives.

3-level strategy:

Level	Method	Requires
1	PRAW (Python Reddit API Wrapper)	REDDIT_CLIENT_ID + REDDIT_CLIENT_SECRET
2	Public API /new.json + /hot.json	None
3	Committed frozen Parquet	None

Official Data Sources — 21 Monitored Sources

#	Source	Category	Primary Method	Fallback	Endpoint
1	InfoMoney	Editorial	RSS	—	infomoney.com.br/feed/
2	Empiricus	Editorial	RSS	Scraping	empiricus.com.br/feed/
3	Money Times	Editorial	RSS	—	moneytimes.com.br/feed/
4	Seu Dinheiro	Editorial	RSS	—	seudinheiro.com/feed/
5	Exame Invest	Editorial	RSS	—	exame.com/feed/
6	CNN Brasil Business	Editorial	RSS	—	cnnbrasil.com.br/feed/
7	Suno Research	Editorial	RSS (Secondary)	—	sunoresearch.com.br/feed/
8	E-Investidor	Editorial	RSS (Secondary)	—	einvestidor.estadao.com.br/feed
9	NeoFeed	Editorial	RSS (Secondary)	—	neofeed.com.br/feed/
10	Toro Investimentos	Editorial	RSS	Scraping	blog.toroinvestimentos.com.br/feed/
11	Funds Explorer	Portal	Scraping	—	fundsexplorer.com.br
12	Status Invest	Portal	Scraping	—	statusinvest.com.br
13	Clube FII	Portal	Scraping	—	clubefii.com.br
14	FIIs.com.br	Portal	Scraping	—	fiis.com.br
15	Portal do FII	Portal	Scraping	RSS	portaldofii.com.br
16	Investidor10	Portal	Scraping	—	investidor10.com.br
17	Eu Quero Investir	Portal	Scraping	—	euqueroinvestir.com
18	Bora Investir (B3)	Institutional	Scraping	—	borainvestir.b3.com.br
19	XP Conteúdos	Institutional	Scraping	—	conteudos.xpi.com.br
20	Investing Brasil	Portal	Scraping	—	br.investing.com
21	Reddit / Google News (Fallback)	Social / Behavioral	PRAW (when available) + Google News RSS (fallback)	r/investimentos · r/farialimabets · news.google.com

Tip

The original behavioral source uses Reddit subreddits (r/investimentos and r/farialimabets) as a social intelligence and market narrative layer.
Following changes to Reddit’s public API policy in April 2023 (HTTP 403 restrictions), the pipeline was redesigned to operate across three levels:

Source 21 — Reddit / Google News (Fallback)

1. Level 1 — PRAW (when REDDIT_API_AVAILABLE = True)

   Uses the authenticated Reddit API to collect recent posts from the target subreddits.

   
   

2. **Level 2 — Google News RSS PT-BR (fallback)*

   • When Level 1 is unavailable (e.g., missing REDDIT_CLIENT_ID in .env or public API restrictions), NB01 triggers collect_google_news_rss(), which:
   • queries Google News in Portuguese using FII-specific search terms,
   • filters content using FII-related keywords (FII_FILTER_TERMS),
   • stores articles with source='news.google.com', source_type='reddit', tags='google_news_rss', and ingestion_method='feedparser_google_news'.
   
   

3. Level 3 — Frozen Parquet (Resilient Snapshot)

   For reproducible evaluations and operational resilience, Source 21 data can be frozen in data/external/ and reused without issuing new requests.

In the documented reference execution, the Google News RSS fallback generated 351 FII-related articles for Source 21, preserving continuity of the behavioral intelligence layer even when direct access to Reddit’s public API was unavailable. [page:46]

🏗️ High-Level Architecture

%%{init:{
'theme':'dark',
'themeVariables':{
'background':'#090d13',
'primaryTextColor':'#F5F7FA',
'lineColor':'#2dd4bf'
}}}%%

graph TD

NB00["NB00<br/>21 DATA SOURCES<br/>RSS • Scraping • Reddit"]:::bronze

NB01["BRONZE LAYER<br/>Ingestion<br/>feedparser • BS4 • PRAW"]:::bronze

NB02["SILVER LAYER<br/>Cleaning & Normalization<br/>Quality Gates"]:::silver

NB03["SILVER LAYER<br/>MapReduce Word Count"]:::silver

NB04["GOLD LAYER<br/>TF-IDF + BM25<br/>Retrieval Index"]:::gold

NB05["GOLD LAYER<br/>Sentiment Analysis<br/>PT-BR Lexicon"]:::gold

NB06["GOLD LAYER<br/>Marketing Intelligence<br/>Signals • Funnel • Insights"]:::gold

NB07["SERVING LAYER<br/>Dashboard Dataset"]:::dash

API["FASTAPI<br/>Serving Layer<br/>REST API"]:::dash

ST["STREAMLIT<br/>Serving Layer<br/>Dashboard"]:::dash

BOT["GROQ CHATBOT<br/>LLM Layer<br/>GPT-OSS-20B"]:::llm

NB00 --> NB01 --> NB02

NB02 --> NB03
NB02 --> NB04
NB02 --> NB05

NB03 --> NB06
NB04 --> NB06
NB05 --> NB06

NB06 --> NB07
NB07 --> API
NB07 --> ST

API --> BOT
ST --> BOT

classDef bronze fill:#2a1512,stroke:#a85a4a,color:#F5F7FA,stroke-width:2.5px;
classDef silver fill:#1b2430,stroke:#b0b7c3,color:#F5F7FA,stroke-width:2.5px;
classDef gold fill:#2a2208,stroke:#e6c35a,color:#F5F7FA,stroke-width:2.5px;
classDef dash fill:#06363d,stroke:#2dd4bf,color:#F5F7FA,stroke-width:2.5px;
classDef llm fill:#231433,stroke:#b56cff,color:#F5F7FA,stroke-width:2.5px;

Loading

Tip

Detailed architecture diagram → docs/architecture.md

4. Project Structure

app/
├── main.py
├── api/
│   └── routes.py
├── services/
│   ├── retrieval.py
│   ├── embeddings.py
│   ├── llm.py
├── models/
│   └── schemas.py
├── db/
│   └── vector_store.py
├── core/
│   └── config.py

5. API Layer (FastAPI)

from fastapi import FastAPI
from app.api.routes import router

app = FastAPI(
    title="Market Intelligence API",
    description="RAG-powered financial intelligence system",
    version="1.0.0"
)

app.include_router(router)

6. Core Endpoint — Semantic Query

@router.post("/query")
async def query_system(question: str):
    
    context = retrieve_context(question)
    answer = generate_answer(question, context)

    return {
        "question": question,
        "context": context,
        "answer": answer
    }

7. Retrieval Layer (RAG)

def retrieve_context(query: str, k: int = 5):
    query_embedding = embed_query(query)
    results = search_vectors(query_embedding, k=k)
    return [r["text"] for r in results]

8. Embeddings Layer

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

def embed_query(text: str):
    return model.encode(text)

9. Vector Store (FAISS)

index = faiss.IndexFlatL2(384)

def search_vectors(query_embedding, k=5):
    D, I = index.search(np.array([query_embedding]), k)
    return [{"text": f"doc_{i}"} for i in I[0]]

10. LLM Generation Layer

def generate_answer(question, context):
    prompt = f"""
    Context:
    {context}

    Question:
    {question}

    Answer:
    """
    return call_llm(prompt)

11. End-to-End Flow

Layer	Function
🥉 Bronze	Raw ingestion and storage
🥈 Silver	Data cleaning and NLP processing
🥇 Gold	Signal generation and ranking
RAG	Semantic retrieval
FastAPI	API interface
LLM	Natural language reasoning

12. Example Query

{
  "question": "What is the current investor sentiment on logistics REITs?"
}

➠ Response:

{
  "answer": "Recent data indicates a moderately positive sentiment driven by stable dividend yields and occupancy rates."
}

Final Note

This architecture transforms a traditional data pipeline into a full-stack AI intelligence system, enabling:

* semantic search
* investor sentiment
* real-time insights
* natural language interaction

How to run this project locally

Prerequisites

- Python 3.10+ installed - Git installed - (Optional) Python virtual environment (venv) to isolate dependencies

Clone the repository

git clone https://github.com/Quantum-Software-Development/5-cybersecurity-social-engineering-fii-marketing-intelligence-platform.git
cd 5-cybersecurity-social-engineering-fii-marketing-intelligence-platform

[Create and activate the virtual environment]]()

# macOS / Linux
python3 -m venv .venv
source .venv/bin/activate

# Windows (PowerShell)
python -m venv .venv
.\.venv\Scripts\Activate.ps1

Note: the .venv/ folder is already ignored in .gitignore, so the virtual environment will not be versioned.

Install dependencies

pip install --upgrade pip
pip install -r requirements.txt

Run notebooks / scripts

• Open the notebooks in the 2-FIIs_Final folder in Jupyter Notebook, JupyterLab, or VS Code.
• Make sure the selected kernel is the .venv virtual environment.
• Adjust data paths if needed (under the data/ directory). Local data layers such as data/bronze, data/silver, and data/gold are git-ignored by default.

Whenever you add or remove dependencies:

pip freeze > requirements.txt
git add requirements.txt
git commit -m "Update project dependencies"

References

• Barocas, S., & Selbst, A. D. (2016). Big Data’s Disparate Impact. California Law Review, 104(3), 671–732.
• Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet Allocation. Journal of Machine Learning Research (JMLR), 3, 993–1022.
• Brasil. (2018). Lei nº 13.709, de 14 de agosto de 2018: Lei Geral de Proteção de Dados Pessoais (LGPD).
• Chapman, P., Clinton, J., Kerber, R., Khabaza, T., Reinartz, T., Shearer, C., & Wirth, R. (2000). CRISP-DM 1.0: Step-by-step data mining guide. SPSS.
• European Commission. (2019). Ethics Guidelines for Trustworthy AI. Brussels: High-Level Expert Group on Artificial Intelligence.
• Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
• Jurafsky, D., & Martin, J. H. (2025). Speech and Language Processing (3rd ed.). Stanford University.
• Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press.
• Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019). Model Cards for Model Reporting. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT) (pp. 220–229).
• Molnar, C. (2022). Interpretable Machine Learning (2nd ed.). Lulu.com.
• Robertson, S. E., Walker, S., Jones, S., Hancock-Beaulieu, M., & Gatford, M. (1995). Okapi at TREC-3. In Text REtrieval Conference (TREC-3). NIST.
• Robertson, S. E., & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4), 333–389.
• Russell, S., & Norvig, P. (2021). Artificial Intelligence: A Modern Approach (4th ed.). Pearson.

💌 Let the data flow... Ping Me !

🛸๋ My Contacts Hub

────────────── ⊹🔭๋ ──────────────

➣➢➤ Back to Top

Copyright 2026 Quantum Software Development. Code released under the MIT license.