TLDR; the repos of the work can be seen on github.
There are lots of vibes when the new year comes and for me it’s trying to focus on something for a little bit, stick with it and get a conviction if it worth continuing to experiments or if it was a nice toy for some days.
I’m working on Business Intelligence using many commercial tools from the database to reporting through ETL and data engineering. Reading and talking more and more about AI and all the fancy stuff we can do with data nowadays make me believe that I should give a real try to Python. I’ve done some tutos, some scripting before but nothing useful for me. As soon as I was « really » working on something, I opened my loved ones as SSIS, Talend or other well-known ETL or tools to process data. The issue with this is that even if there is connectivity, capability to do scripting it’s always within the boundary of the tool and what it’s editor permits.
Frustrated and being constantly pushed to try new things I started to go a bit deeper with Python. Three main reasons for this:
- It’s a general-purpose language –> I’ll be able to do whatever I want as long as it relates to programing. And it will be useful for later experiments.
- Python is broadly used for IOT (I hope MicroPython is not to Python what JavaScript is to Java )
- Python is becoming the De Facto language for data science (well, yes there R in the picture but seems too specialized for me).
Enough history, what have I built ?
The use case is somewhat classical and I really thought I would have found something ready on the Internet but my research didn’t get it. I need to extract data (a lot) from an Org using the Microsoft Graph API in bulk but with no right to register an app! Every damn bit of code I found need to register an app in the AAD of the Org to get an app key to be able to query the graph api. In my case, in my company there is no way I can get such permission and it’s also a good catch if I can get rid of this in order to be « free » to run my code with no previous configuration.
In a series of articles, I’ll try to give you my journey on learning how to use Python in an effective and easy way and what tools I crossed and how they changed my view on the dataengeneering landscape.