Real-Time Data Analytics

The Prime View had the privilege to speak with Mark Palmer, Senior Vice President, General Manager of Analytics, Data Science & Data Virtualization at TIBCO.

Read the interview to learn about M&A as a good method to bring diversity to your organization, aligning business leadership with personal values, and the reasons why TIBCO prioritizes contributing to Kafka.

Mark, can you talk a bit about your background leading up to your current position at TIBCO? 

I started off as a developer and eventually wound up with startups and entrepreneurial efforts in Boston. I linked up with a company called StreamBase founded by Michael Stonebreaker. Dr. Stonebreaker is well known in the database industry as a computer scientist specializing in database research. I joined him as the StreamBase CEO and eventually sold that company to TIBCO, another real-time streaming pioneer.

How did the acquisition change your approach to building products and innovations in general? 

When you’re an independent, fueled by VCs company, you’re free experimenting. It’s a natural startup culture where you’re going to fail and try again until you finally find the market and then grow. 

Big companies, on the contrary, tend to have a bit of an immunity to failure. Also, a big company has its advantages of scale. And sometimes, that comes with a downside. However, at TIBCO, the culture is very much tolerant to failure and experimentation, and growth. We have good leadership, and it’s fun to work there.

Mark Palmer

SVP & General Manager at TIBCO

That begs a question about your view on the qualities of an effective leader. How can one become an effective leader?

I think it would be quite arrogant to say, ‘I’m a good leader,’ or ‘Here’s a recipe of how you become one.’ I don’t know that.

I compare leadership to the elements of being a good dad or a good husband. At least in my way of doing things, it’s being a good listener, being humble, and learning from the people around you. 

I blog about many things, and one of the things I talked about was becoming a widow when my kids were young, when they were six and four. That was a big shock because I had to be the emotional support and the leader at home all of a sudden. So, that is somewhat similar to my work, where I have to support and lead teams. 

As you know, at TIBCO, we’re at the forefront of using data and analytics in real-time, and I think it’s essential to be excited about the company’s vision and share that excitement with your team. So, I think that’s my job as a leader is to be excited about that vision and describe it to the team so they may get some of the enthusiasm.

I’ve subscribed to the blog and started reading it from your personal story about becoming a single parent when you lost your wife. I was impressed with the part where you compared difficult times to an emergency landing on a plane – you need to take care of yourself first and then help kids deal with the circumstances. Do you see any parallels between managing crisis at home and at work?

I believe there are many parallels between being a good parent and good at work, especially in terms of a diversity of views. As an example, for me, the most challenging thing with my daughter is to see things from her point of view. Oftentimes, that translates to my work with teams. 

What types of individuals you’re looking to bring to your organization to establish that innovation culture? 

At TIBCO, diversity is a big theme. That might be not only gender diversity or ethnic but also cultural diversity. As you may know, we have a distributed team throughout the world – big centers in Sweden and Vietnam, which are both exciting places culturally to learn about and to work with. This diversity gives us new perspectives to look at our work. 

To establish the innovation culture, we try a vibrant, internship-type program to get some new perspectives and new thinking ways. Personally, I spend a lot of time talking to new engineers because they’re looking at our products with fresh eyes.  

M&A work well for diversity, too. We’ve acquired six companies in the last five years and got a great variety of thought from them. One of the acquisitions was a French company whose engineering organization was in Vietnam. They had forty-five percent of women in Vietnam’s engineering team, and we’ve learned a lot from them when they came into our company. It’s fascinating how you can grow your own culture through acquisitions. Often, people think of it the other way around – the big company imposes its culture on a little company, but we’ve learned from them in that case.

Mark, you were into streaming technology a long time before it became popular. What sparked your interest in this specific technology?

There were several things that sparked my interest in streaming technology. Back in the early 2000s, RFID (Edt. Radio Frequency Identification) was one of the hot tech topics. The tags used to track Walmart’s shipments weren’t as cost-effective as they are now. But the wide use of the RFID tags by Walmart gave the technology the necessary development thrust into the mainstream. Back then, you could never imagine putting a tag on a can of soup because the tags themselves cost fifty cents, and that’s the margin on the canned soup. If you look at it now, you won’t get any delivery without a scan or a tag connected to a sensor. So, that was one of the points that sparked my interest. 

Another one was joining StreamBase. There was a bunch of academic research there at the time – John Bates at Cambridge University, Dr. Michael Stonebreaker at MIT, whose students like Richard Tibbets, Eddie Galvins were doing cool thinking in the early 2000s and then spawned their companies.

The whole idea about streaming technologies is the shift in computing you have to go through when data is in motion. It’s a fundamental change. And that’s what got me most excited about it.

Now with IoT, everything’s connected. Our kids are connected, our refrigerators are connected, and light bulbs are connected too. It’s a kind of a computing playground. And, of course, with open-source, cloud-enabled technology, the entire industry is connected and evolving rapidly.

Customers demand real–time, highly accurate, streaming analysis of workflow data so they can make rapid business decisions or respond to a disaster. It’s trending technology now.

It’s impressive too because at the same time it’s very old. I mean, if you trace it back, it was first used in planes, in the wars back in the 40s and 50s, and the tags were probably the size of a refrigerator at that time. But with miniaturization, with the shift to the cloud, and 5G, these things make it much easier to compute with connected things.

In the video, you speak about Kafka as a ‘de-facto’ standard for enterprise messaging. I’m wondering what is TIBCO’s approach to open-source software (OSS)? Do you contribute to projects like Kafka? Does it help you to accelerate the pace of building innovative products?

It’s a great question. I did say ‘de facto’ for Kafka because open source is a tsunami, and everybody reaches for it. And TIBCO, of course, supports that. We distribute and contribute to Kafka; we’re one of Kafka’s leading providers in the industry. Kafka is interesting because it has a wide range of messaging in real-time systems. I like the old saying here at TIBCO, “If our messaging doesn’t work, then planes don’t fly, transactions don’t happen on Wall Street, and FedEx doesn’t deliver packages.”

So, when you get up into that kind of mission of critical messaging, there’s no substitute for some of the advanced things that TIBCO does – we have the whole range of technologies from open-source to mission-critical messaging. And with the rise of IoT, other dimensions are critical, too. MQTT has different variants of messages that are specifically designed for edge computing and IoT data streams.

Real-time messaging is a gigantic field, and one of our significant areas of concentration is to make Kafka better.

We think that Kafka is excellent if I could compare it with plumbing – plumbing is essential to your house, but at the same time, you don’t want a plumber to design your kitchen. This analogy might help you understand cloud infrastructure. On top of that, you have to put analytics visualization tools that make it easier to deal with the streaming data. 

Answering directly to your question, we support Kafka. We love it, contribute to it, extend it, and try to make it better and democratize Kafka to make it accessible to everyone.

You’ve started talking a bit about IoT and connected devices and their impact on demand for streaming technologies. In your opinion, how IoT and 5g will impact the demand for streaming technologies? 

You can’t do streaming analytics until you have a stream. The real-time streaming data took about twenty years to hit that tipping point. 5G is now just coming. When they first came out, sensors were way too expensive to put on retail items or household goods. Now, it’s kicked in, and it is cost-effective. 

5G is effectively like the dial tone, but the dial tone didn’t stop. We could have kept going with mobile over IP. To me, 5G and Kafka, and networks and sensors – all that stuff is just the plumbing that makes it easy to access those real-time data. 

The exciting thing is that, even though the technology is relatively mature, only about 11% of companies have access to a database that’s less than an hour old, meaning they don’t have real-time data or voluminous real-time data. There’s still a long way to go. And that creates an opportunity for innovation—fun to be at, for a technologist.

Could you elaborate on the goals of TIBCO LABS? What are the most exciting projects or ideas that originated from the department?

We wouldn’t refer to it as a department. I think of it as the whole company’s job is to be a lab for innovation. If you’re in the tech industry, you’re always innovating. Nelson Patrick, CTO at TIBCO LABS and leads the experimental work, while I’m the guy who commercializes the products. It’s all one team. We’re intimately involved with TIBCO LABS. 

There’re lots of examples of the cool things that I get excited about. Last year, we issued many patents that came out of either joint work with the Labs or purely out of engineering.

One of the exciting projects we’ve done is Dynamic Learning, a machine learning technique that’s more popularized in the space of teaching robots how to walk on their own. It’s incremental learning, a technology that learns on streams instead of just training on historical data; your training is based on what’s happening right now. And that’s an essential improvement for environments like high tech manufacturing, where things are continually changing – you’re changing the machine setup, configurations are different, the factors that might predict breakage or loss of yield are going to change over time. 

The Dynamic Learning model learns as the system changes on the fly. And that’s important when you don’t have so much big data, and it’s screaming at you all the time. That’s something that we issued a patent on last year that I’m very excited about.

We’ve done other things in streaming through Labs, a project we think of as a digital DVR. The DVR principle is essential in the streaming data world. For example, an oil rig could suddenly break down, but with our technology, you could replay the events that led up to the problem by replaying all those data points. That’s a real breakthrough innovation. 

The last one that springs to my mind is a radical simplification idea that we got about a wizard that you attach to Kafka, and it learns what data types are on the stream and makes it with no code connected to a Visual Dashboard. 

As you may see, I’m fascinated by the innovations that make your job easier. That’s the way we ultimately democratize and popularize all this real-time stuff.

Mark, thank you for the interesting conversation and for showing how business values may align with your own personal values.

Stay tuned for the next interviews!