Why do you need a technology architect?

Navigating technology is hard, even for seasoned professionals.

I've got a passion for finding solutions for complex problems, particularly when it benefits someone or some group I'm invested in emotionally.

It's difficult to know which technology is the correct choice, given each business has it's own nuances, and the options are are seemingly endless.
Technology selection may seem like a simple research issue at first; and in some cases where the risk or consequence of failure is low, it may be an easier task.

We'll assume in this article you have an interest in growing a business, and potentially have a vested interest in the business success.
We'll also assume that your business uses software to manage data relevant to your business or customer's experience.
I'd be surprised if these assumptions don't apply to most businesses.

Where to start

One problem with technology selection and solution engineering, is knowing where to start and what's available. With the quantity and speed at which new technology is released, there's usually a delay in the time it becomes available and the time we "discover" it.
Search engines help with this - but many times, you need to first know what to search for. Or you might start with a general research and start narrowing down ideas, eventually delving into the deeper topics. This can consume hours, days, weeks, or even longer depending on how much information you need to make a decision. Staying up-to-date with new technology and trends is practically a full-time job. More so now that AI is rapidly evolving, pervasive, and tools are released more frequently.

How do you know what to search for?
How do you identify which technology to combine for a full solution?

Spending a few minutes drawing a mind map of the general technology domain quickly becomes unmanageable in a 2 dimensional area, and would be better represented in a 3 dimensional graph. The image below, depicts a very high-level relationship between some concepts, and only exposes a small fraction of the domain considerations for a full business solution.

Other questions we might ask when planning:

Will the solution infrastructure scale?
Does it scale financially with your growth?
How long will this solution work - what's the likelihood of deprecation?
What do the day 2 operations look like and how much does maintenance cost?
Will you be locked in, or will it take significant work to migrate/upgrade?
What does the security look like and can you delegate responsibilities safely?
What options are available for integration?
Any many more...

LLMs may be the next evolutionary step in the research process, and I've been experimenting with this for research lately. Unfortunately, there are issues around hallucinations where the model may return a realistic answer although it may be wildly inaccurate. I suspect this will improve significantly over the next couple of years.

If the problem space is so large and complex, how could anyone stay completely up-to-date?
Maybe it's not possible - but some of us are obsessed with learning and experimentation, and I believe this is a common attribute of the best architects. Software projects and businesses fail for a variety of reasons, but in my experience, selecting the wrong technology or having a poor architecture can be at best costly, and at worst fatal to the business.

Professional Geek

Here's a little context of my background.

I'm a maker at the core, whether it's music, art, food, software, or anything else that allows me to be creative and bring something new into the universe.

I've been fascinated with computers since the 80's, and it would've been sooner had I been born earlier. Since then, I've been pumping my brain with technical documents, research, proof of concepts, etc...

The first computer software I "wrote" was in the early 90's on an Amiga 500 using AmigaBASIC. I think I was in 6th grade, when I'd found a script in the back of a math book. Although I didn't care too much about learning algebra and turning homework assignments in, I suddenly found this schoolbook interesting.
After carefully copying the code from my book into the Amiga, which was probably less than 100 lines, I executed the script and behold! There were a random number of stars printed on the screen, and I could play the game by entering a guess of the star count on the command line.

Over the next 10 years or so, I experimented with other languages like Perl, VBA, and C. I started learning more about HTML and JavaScript. The inspiration for learning web development was firstly because I needed a page to recruit for my gaming clan (anyone remember Tribes?).
Secondly, while doing tech support for Bell Atlantic DSL, I created a web page to organize all my favorite links. Other tech-support reps liked my "cheat-sheet" so I emailed it to them, and I was promoted to a mentor role to assist others.
This may have been the first time I considered development as a way to make money, but I didn't know how. I also didn't know the term "knowledge delivery" and that I'd started a journey with that simple web page of links that would be a driver in my career.

Geek tip -
Take time to help your family and friends when they ask for help with their electronic devices, no matter how far you climb the tech-ladder.

Open source in the enterprise

Over the next decade, I'll just mention that I worked with many, many, different frameworks and languages, and eventually became the architect of a learning management system and a mission critical support platform for a large telecom enterprise. "Knowledge delivery" was at the core of these systems and I was fortunate to meet some of the leading engineers of search related technology at Lucene conventions, got to play with UIMA when IBM was developing Watson, and learned much about relevancy tuning and machine learning.
My employer was great - giving me freedom to find problems and create solutions, with few directives from management.
I found that I could have the job I wanted, by looking for the opportunities to use the tech I was interested in, and creating a proof of concept solution.
It only worked some of the time, but I gained so much insight and experience by repeating the try/fail/try cycle, that I became very fast at exploring new ideas and technology. So that when it did work, the value was great.

With this approach, I was able to deploy cutting edge solutions to introduce our development team to:
- Low code tool development (EMF, Papyrus)
- Package management (Maven, Artifactory)
- Continuous integration and deployment automation (Jenkins)
- Site reliability engineering (CollectD, Graphite, scripting, and later Grafana)

We reduced our painful 3-5 hour deployment meetings with 10-20 people on a conference, to about 15 minutes with 3 people.

This was circa 2010, so the technology landscape has changed drastically since.

A changing landscape

Today we use virtualization at multiple levels, containerization, and layers upon layers of automation to deploy full technology solutions.

With the applications and tools available today, particularly in the open source space, a single person today can produce as much as an entire team did a few years ago, given the correct architecture and technology selections.
I can confirm this from my own experience, and have architected, implemented, and supported digital infrastructures for companies at a fraction of the cost previously required.

Horizontal/vertical autoscaling help reduce cost by only allocating resources when they're required.
CICD reduces time/money for testing and deployments.
Frameworks like Apache Causeway reduce time/money required to implement custom business functionality.
Event driven designs and supporting frameworks / micro-saas enable flexible change, reducing total cost of ownership.

For one of Savantly's clients - we architected and implemented an enterprise solution, including an oauth identity provider, data lakehouse and warehouse, ELT pipelines, a Spark cluster, a Kafka cluster, a custom core business application, reporting and analytics services, digital asset management, internal websites, and a public website that has spikes to 1 million hits in a day, for less than $3k/month avg in infrastructure - all owned by the client.
If you look at enterprise level commercial SAAS offerings for that platform, you'll likely pay 3-5X the amount, and potentially much more for multi-site businesses.

We can solve your technology woes

If you need help navigating your cell phone, I encourage you to contact the geek in your family or social circle.  =D
If you need help selecting technology for your business or architecting a solution, contact us to schedule an introduction!