Beyond the Hype: Maximizing the Impact of AI Development Tools
Unlocking Potential While Navigating Limitations
AI development tools are revolutionizing software creation, empowering developers and non-developers alike to build applications faster and smarter. Yet, as with any innovation, they come with limitations that demand careful navigation. Here’s a deep dive into what AI development tools do well, their limitations, the opportunities they unlock, and best practices to follow.
AI Development Tools Overview
The number of AI enabled development tools continues to rapidly expand. As outlined in Lenny's newsletter, they fall into 3 broad categories:
Standalone AI assistants can write and explain code, perform data analysis but usually require a companion tool to run the code generated.
AI-enabled development environments (IDEs) provide fully integrated development environments to build applications in any number of programming languages with embedded AI assistants or agents.
Full stack AI agents provide a full stack application development environment from development to deployment in a cloud environment.
A Deeper Look at Standalone AI Assistants
Best Suited For:
Standalone AI assistants are best suited for code reviews, data analysis and visualization, or simple one-page applications. These tools will generate code to support a request. You can use them to build pieces of an application, but they will quickly run into limitations as your request increases in complexity. While standalone AI assistants can run some of the code they generate, they typically require another tool to run the generated code.
Vendors:
ChatGPT and Claude are best suited for code and data analysis. Perplexity is best suited for research and can also generate supporting code.
Google Gemini and Microsoft Copilot provide the best integration with authoring suites. Gemini and Copilot are also getting bundled with Google Workspace and Office 365 if you use a paid subscription. Both Microsoft and Google have started to include Gemini and Copilot in their entry level plans.
Usage Examples
Competitive Research
Perplexity shines for this type of use case, its space functionality enables you to create agents to support your research needs. You can provide specific system instructions to narrow the scope of your space / agent and templatize its response. Perplexity also lets you choose the LLM model to work with providing a broader base for comparing responses.
The figure below illustrates how a Perplexity space can be configured for researching vendors in a domain, specifying space instructions, choosing the AI model and resulting into a generated Python output.
Figure 1 - Using Perplexity Spaces for Vendors Research
Financial Data Analysis
For advanced data analysis, Claude is a very effective model. Provided a detailed prompt, Claude can rapidly produce a model that analyzes 100s of permutations of possible investment allocations, model the portfolio return and risk profile, provide allocation recommendations, default visualization and the Jupyter notebooks for future analysis. In this instance, Claude essentially acts as your personal data analyst. To fully taking advantage of Claude's analysis, you need to use supporting tools like an IDE for the generated code execution, data science tools like Anaconda for running the Jupyter notebooks. Claude can be used as professional coach, dramatically accelerating access to advanced skills required for data analysis. Using Claude enables you to take advantage of tools and techniques difficult to access and master on your own.
The figure below illustrates the generated Python project structure that Claude created, including a fully functional visualization of portfolio options and Jupyter notebooks to be used for further analysis.
Figure 2 - Using Claude for Investment Modeling with Project Structuring, Options Visualization and Jupyter Notebooks
Code Analysis, Documentation and Development Best Practices
Provided with code to analyze, ChatGPT or Claude will make recommendations on what to improve, where to add documentation and will generate the documentation supporting your project.
In the figure below, Claude was asked to review and document a Model Context Protocol (MCP) command line interface (CLI) implementation. Claude generated recommendations for improvement areas and the documentation for the MCP CLI.
Figure 3 - Using Claude for Code Review and Documentation
A Deeper Look at AI-enabled IDEs
Best Suited For:
These tools are best used for standalone development. They have 2 very different audiences:
Non-developer audience for whom they dramatically simplify access to an integrated environment required to run code. While they might look intimidating to the non-developer, the best tools enable you to run generated code with very little programming skills. They provide an excellent complement to AI assistants code output.
Developer audience for accelerated code creation and development in most common languages and frameworks.
Vendors:
This area is experiencing a significant proliferation of tools. Many of these tools are based on a fork of Visual Studio Code open-source core and extended with their own AI assistants or agents:
Cursor is the most prominent AI code editor with seamless integration of an AI-based chat used for interacting and updating code using natural language.
Windsurf and GitHub Copilot are other popular options.
Usage Example:
Non-developer Code Assistant
As full fledge IDEs, these tools can be used for any level of development from simple to complex projects.
The example below focuses on how to use Cursor as companion tool to an AI assistant. Cursor simplifies running Python code created by the AI assistant, making it accessible even for non-experts. In the example below, a research AI assistant generated multiple research outputs in the form of comma-separating files. It also generated a Python script to merges these comma-separated files and remove duplicated entries. Cursor is used to run the python script and get a final consolidated output. The Cursor AI chat helps the user troubleshoot issues that might arise running the python code.
Figure 4 - Using Cursor for Guided Code Execution
A Deeper Look at Full stack AI Agents
Best Suited For:
These tools are becoming incredibly powerful for rapidly prototyping fully functional applications in just a few hours. Most of these tools provide similar capabilities to AI enabled IDEs but include full runtime and cloud deployment options. This enables you to easily run and deploy your application and to make it available to external users.
Vendors:
Many vendors are entering this rapidly evolving space:
Replit provides a robust multi-agent application building environment
v0 is building a reputation for building visually appealing applications
Lovable exclusively focused on building apps via prompting. Unlike other options, it does not provide a code editor.
Usage Example:
Fully Featured Prototype Creation for Expense Tracking
Using Replit, you can rapidly implement a fully functional application. Below are the requirements of an expense tracking application I developed in an afternoon for $11.50. I started with a basic set of prompts and iterated with the agents for several hours, all the code produced was written by the Replit agent. While Replit provides a monthly subscription, it fully meters your usage of their tools as you build and run your application. This provides full visibility into your application development cost.
Figure 5 - Expense Management Application Requirements
Below is a short video of the Expense Tracker application created.
Figure 6 - Expense Management Application Built and Running in Replit
The Power of AI Development Tools
AI tools are becoming indispensable for research, prototyping, data analysis and modern software development, offering significant advantages:
Automation of repetitive tasks: AI tools streamline research, analysis and coding by automating mundane task with code generation, debugging, and testing.
Enhanced productivity: By reducing manual intervention, AI tools make powerful and advanced techniques available to non-developers and allow developers to focus on complex and creative tasks.
Improved accuracy: AI algorithms excel at identifying bugs and optimizing code. AI agents ensure higher-quality outputs by analyzing patterns and suggesting improvements.
Faster prototyping: Platforms like Replit enable rapid prototyping by turning ideas into functional applications in minutes. This accelerates innovation and reduces time-to-market.
Democratization of development: AI tools make software development accessible to non-technical users, broadening participation in the tech ecosystem.
Seamless integration: Advanced AI tools integrate with existing development environments (e.g., Visual Studio Code), ensuring smooth workflows without disrupting current processes. Standards like Claude Model Context Protocol connect AI assistants to the systems where data lives, including content repositories, business tools, and development environments.
The Limits of AI Development Tools
Despite their benefits, AI development tools are not without challenges. It is important to be aware of these challenges to fully take advantage of these tools.
Output Fidelity and Code Quality
AI tools can struggle with output fidelity and experience code quality challenges:
Output fidelity: when using AI tools for prototyping, it might be challenging to get the right level of output fidelity. Getting that extra 5% in fidelity can turn out to take significant effort. When running into such issues, it is important to remember and refocus on the true intent of the prototype.
Quality challenges: AI tools "love" to rewrite code. They might also not be consistent in their coding patterns. For instance, imagine having a configuration service that you want to consistently use for centralizing all your configurations. Without explicitly instructing the AI to use this service for all the configuration aspects of your implementation, the AI might skip using it at times even though it created it. Developers should provide explicit instructions or use global settings to ensure consistency in generated code. Tools like Cursor provide global or project level system instructions for their AI agent to comply with specific rules. It is important to know how to use these techniques to put constraints on AI tools and to drive the right behavior. You should also always review and understand the changes being made by the AI tool.
Context challenges: as the complexity of your task or application increases, AI development tools will often struggle to maintain the context of the task (e.g. complex data analysis or full code base) periodically requiring you to explicitly restate the context.
Critical Thinking and Risk Management
When you no longer understand what an AI tool is doing and why, that should be a serious red flag:
Over-reliance on AI: the more you use these tools, the easiest it is to become overly dependent on them. For instance, these tools will recommend additional features, quickly leading to feature bloat. They make interesting recommendations on what you can add to your application. But like in the real world, more features mean more complexity and with more complexity, the AI tools become less performant. So, it is important to remain focused on what you want to build. For more developer centric tasks, developers risk losing critical problem-solving skills by becoming overly dependent on AI tools. Like functional bloat, code bloat is real with AI tools and adds unnecessary complexity.
Outdated dependencies and security risks: most models are current up to a specific point in time. As a developer, it is easy to lose track of that. For instance, Claude Sonnet 3.5 released in October 2024 is current up to April 2024 (yes, even for the upgraded version). As a result, it may not be aware of security vulnerabilities introduced after April 2024. You should never trust that AI generated code is secure. If you are going to use AI beyond prototyping, you need to use mature code scanning techniques like static and dynamic code analysis tools (e.g. SonarQube, Snyk, Veracode) to vet your code's security.
Maximizing your Impact with AI Development Tools
The first experience with AI development tools is almost magical. These tools are reshaping the software industry by boosting productivity, enhancing accuracy, and democratizing access to development capabilities.
Best Practices
While powerful, these tools have clear limits as well. It is important to understand why and how to work within those limits. To maximize your impact with these tools:
Experiment, use different tools, understand their strengths and combine them for maximum impact
Always remain in the driving seat and understand what the AI tool is doing:
For non-developer, stay focused on your goals and keep control of the scope.
For developers, always understand what the AI tools is doing. Always maintain a clear view of your components' architecture and interactions.
Never just trust the AI. This is particularly true in areas like security where the AI tools knowledge will almost always be outdated.
Keep your critical thinking: these tools are a blessing if used right, they enable us to elevate our thinking and focus on higher level concerns. Used poorly, they can become a substitute for thinking and produce undifferentiated outcomes.
Figure 7 - Maximizing AI Development Tools Impact
Call to Action
If you are a business analyst, product manager or product designer, these tools will change the way data is analyzed, product are created, tested and defined. They enable rapid prototyping and testing of ideas. They break the barrier between requirements (product manager), designs (designer) and working prototype (developer). Challenge yourself to build prototypes.
If you are an engineer, you can supercharge your productivity. It creates opportunities for testing more solutions options, making sure you deliver the most effective solution to your customers.
By following best practices with these powerful new tools, non-developers and developers alike can push the boundaries of innovation.