Hacking MetaGPT

MetaGPT bills itself like this:

Takes one line requirement as input and outputs user stories, competitive analysis, requirements, data structures, APIs, documents, etc.

Its slogan is Code = SOP(Team): encode the standard operating procedure of a software team (product manager, architect, project manager, engineers) and let a crew of LLM agents run it. I had to see what it actually produced.

Setup

git clone git@github.com:geekan/MetaGPT.git
brew install npm
sudo npm install -g @mermaid-js/mermaid-cli
python3 -m venv venv
source venv/bin/activate
pip install -e .
cp config/config.yaml config/key.yaml
python startup.py "Build me a hello GPT website"

The first run

I asked for a simple hello-world site. About five minutes later I had something much bigger: a full set of documents and a half-built interactive GPT wrapper. The agents generated a PRD with user stories, a competitive analysis (against the GPT-3 Playground, ChatGPT, DeepAI, Hugging Face, and InferKit), a system design targeting Flask and Bootstrap, a task breakdown, and API specs, each with Mermaid diagrams. Total cost for the run: about $0.22 on a 3,000-token budget.

The generated tree:

.
├── docs/
│   ├── api_spec_and_tasks.md
│   ├── prd.md
│   └── system_design.md
├── gpt_interactive_web/
│   ├── main.py
│   ├── static/
│   │   ├── script.js
│   │   └── style.css
│   └── templates/
│       └── index.html
├── requirements.txt
└── resources/

Getting it to run

The generated requirements.txt didn't install: it pinned bootstrap==4.6.0, which isn't a real PyPI package. I installed Flask and transformers by hand, and PyTorch too, since the app used GPT2LMHeadModel.

The main.py was a small Flask app that loaded a model in its constructor and exposed a POST /generate route taking JSON, running the GPT-2 tokenizer and model, with error handling around both load and generation.

The result

After PyTorch went in, the endpoint actually responded:

POST /generate   {"input": "tell me a joke"}

The output was nonsense, the same phrase about not telling a joke repeated about ten times. Less "comedian," more "this thing's been drinking and it's late."

The documentation generation is legitimately impressive. The working software at the end of it, less so. Worth knowing where the line currently sits. Until next time, sk out.