Zero To Pandas

Peta Gunatanah Malaysia 2014 - 2018

Peta Gunatanah Malaysia 2014 -2018 ("Malaysia's Land Cover 2014 - 2018") web application is a platform generated for the Quality Assessment activity organized by Forest Research Institute Malaysia (FRIM) on 23rd June 2024.

The workshop aims to collect field/reference data from Malaysian's state agencies in the effort to verify the quality of the land cover classification output generated in support of CO2 release measurement from converted agricultural lands.

Participants are able access the app via conventional browsers from their mobile devices and submit drawings/sketches that they have captured within interactive data layers.

This web app aims to support direct input from source onto the task of improving the accuracy of the generated land cover maps. Vectors generated from this exercise are readily standardized with the required data scheme from quality assessment, making full use of the ArcGIS Online ecosystem full to a produce concrete output and actionable information.

🗻 Diorama Drama

So, John Nelson made this absolutely simple tutorial on how to create a diorama interesting features on Earth's surfaces. I have been eyeing alot of people making them and resisting the urge to make one. I mean, I have the habit of falling into a rabbit hole with making maps that I'll definitely be having too much fun to stop. That could mean hours and hours scrutinizing colors, perspective or setups and even legend arrangements. But this...I decided not to overdo it and just make one.

I can't believe how easy it is to make one! Here's a piece of many to come; the elevation diorama of area within the Batang Ai Dam and Batang Ai National Park in Sarawak, East Malaysia.

What you'll learn... 1️⃣ How to extract raster from existing image layer in Living Atlas; that's how we extracted the DEM layer for this elevation data.

2️⃣ How to create a 3D scene in ArcGIS Pro and navigate through them.

3️⃣ How to create a layout layer and export them as a static image format i.e jpeg, png, tif etc...

Check out the video tutorial here 👇🏻

I was thinking that this tutorial is making me feel like there is a possibility of creating some floating island or pixel-like models of isolated pieces of land -- my head is imagining some kind of dragons flying around in some nonexistent Viking fantasy map. But hey...if you're not inspired to create crazier maps from John's tutorial, then are you even a cartographer at all?

He's on Twitter, LinkedIn, YouTube and he even has this supercool blog of his; Adventures in Mapping, where he shares all of his tutorials in full documentation and gorgeous portfolio that could only consist of magic! One such as I can only aspire.

So, that was what our Sunday looked like and I think I'll be playing around with dioramas for a few weeks. Let's see just how many of these I can make 'til the next Sunday.

Python: Geospatial Environment Setup (Part 1)

Here’s a quick run down of what you’re supposed to do to prepare yourself to use Python for data analysis.

Install Python ☑

Install Miniconda ☑

Install the basic Python libraries ☑

Create new environment for your workspace

Install geospatial Python libraries

🐍 Installing Python

Let’s cut to the chase. It’s December 14th, 2021. Python 3 is currently at 3.10.1 version. It’s a great milestone for Python 3 but there were heresay of issues concerning 3.10 when it comes to using it with conda. Since we’re using conda for our Python libraries and environment management, we stay safe by installing Python 3.9.5.

Download 👉🏻 Python 3.10.1 if you want to give a hand at some adventurous troubleshooting

Or download 👉🏻 Python 3.9.5 for something quite fuss-free

📌 During installation, don’t forget to ✔ the option Add Python 3.x to PATH. This enables you to access your Python from the command prompt.

Installing Miniconda

As a beginner, you’ll be informed that Anaconda is the easiest Python library manager GUI to implement conda and where it contains all the core and scientific libraries you ever need for your data analysis upon installation. So far, I believe it’s unnecessarily heavy, the GUI isn’t too friendly and I don’t use most of the pre-installed libraries. So after a few years in the darkness about it, I resorted to jump-ship and use the skimped version of conda; Miniconda.

Yes, it does come with the warning that you should have some sort of experience with Python to know what core libraries you need. And that’s the beauty of it. We’ll get to installing those libraries in the next section.

◾ If you’re skeptical about installing libraries from scratch, you can download 👉🏻 Anaconda Individual Edition directly and install it without issues; it takes some time to download due to the big file and a tad bit longer to install.

◾ Download 👉🏻 Miniconda if you’re up to the challenge.

📌 After you’ve installed Miniconda, you will find that it is installed under the Anaconda folder at your Windows Start. By this time, you will already have Python 3 and Anaconda ready in your computer. Next we’ll jump into installing the basic Python libraries necessary for core data analysis and create an environment to house the geospatial libraries.

📚 Installing core Python libraries

Core libraries for data analysis in Python are the followings:

🔺 numpy: a Python library that enables scientific computing by handling multidimensional array objects, or masked objects including matrices and all the mathematical processes involved.

🔺 pandas: enables the handling of ‘relational’ or 'labeled’ data structure in a flexible and intuitive manner. Basically enables the handling of data in a tabular structure similar to what we see in Excel.

🔺matplotlib: a robust library that helps with the visualization of data; static, animated or interactive. It’s a fun library to explore.

🔺 seaborn: another visualization library that is built based on matplotlib which is more high-level and produces more crowd-appealing visualization. Subject to preference though.

🔺 jupyter lab: a web-based user interface for Project Jupyter where you can work with documents, text editors, terminals and or Jupyter Notebooks. We are installing this library to tap into the notebook package that is available with this library installation

To start installing:

1️⃣ At Start, access the Anaconda folder > Select Anaconda Prompt (miniconda3)

2️⃣ An Anaconda Prompt window similar to Windows command prompt will open > Navigate to the folder you would like to keep your analytics workspace using the following common command prompt codes:

◽ To backtrack folder location 👇🏻

◽ Change the current drive, to x drive 👇🏻

◽ Navigate to certain folders of interest e.g deeper from Lea folder i.e Lea\folder_x\folder_y 👇🏻

3️⃣ Once navigated to the folder of choice, you can start installing all of the libraries in a single command as follows:

The command above will enable the simultaneous installation of all the essential Python libraries needed by any data scientists.

💀 Should there be any issues during the installation such as uncharacteristically long installation time; 1 hour is stretching it, press Ctrl + c to cancel any pending processes and proceed to retry by installing the library one by one i.e

Once you manage to go through the installation of the basic Python libraries above, you are half way there! With these packages, you are already set to actually make some pretty serious data analysis. The numpy, pandas and matplotlib libraries are the triple threat for exploratory data analysis (EDA) processes and the jupyter lab library provides the documentation sans coding notebook that is shareable and editable among team mates or colleagues.

Since we’re the folks who like to make ourselves miserable with the spatial details of our data, we will climb up another 2 hurdles to creating a geospatial workspace using conda and installing the libraries needed for geospatial EDA.

If you're issues following the steps here, check out the real-time demonstration of the installations at this link 👇🏻

See you guys in part 2 soon!

diy geo app

Esri has been releasing more and more MOOC over the span of 2 years to accommodate its increasingly large expanse of products within the ArcGIS ecosystem. 

But of all the MOOCs that I've participated in, 'Do-It-Yourself Geo App MOOC' must be the most underrated ones produced by Esri Training. The functionalities highlighted within the MOOC took the anthem right off their recent Esri UC 2020 that went virtual. The curriculum includes:

The creation of hosted feature layer (without utilizing any GIS software medium like ArcMap or ArcGIS Pro).

The basics of the ArcGIS Online platform ecosystem:

hosted feature layer >  web map > web app 

Basically, to view a hosted feature layer, you will need to drag it onto a 'Map' and save it as a web map.

Conventionally, web map suffices for the visualization and analytical work for the likes of any geospatialist who are familiar with Web GIS. 

But this time, Esri is highlighting a brand new web map product called 'Map Viewer Beta'. Why beta? Cause it is still in beta version but so sleeky cool that they just had to let every have a shot at using it. Truth be told, Map Viewer Beta did not disappoint.

Even so, Map Viewer Beta still has some functionalities that have yet to be implemented. 

Using web map to visualize data, configure pop-up, execute simple analysis and extending it to Map Viewer Beta interface 

Utilizing Survey123 for crowdsourcing data; the first level of citizen science and creating a webmap out of it.

Creating native apps using AppStudio for ArcGIS; no coding required. 

Some tidbits on accessing the ArcGIS API for JavaScript

I love how cool it is that this MOOC actually shows you step-by-step on how to use the new Map Viewer Beta and explain the hierarchy of formats for the published content in the ArcGIS Online platform

I have established my understanding of ArcGIS Online ecosystem 3 years back but I do find it awkward that such powerful information is not actually summarized in a way that is comprehensible for users that have every intention of delving into Web GIS. And Web GIS is the future with all the parallel servers that could handle the processing/analysis of large amount of data. ArcGIS Online is a simplified platform that provides interfaces for the fresh-eyed new geospatial professionals. 

It is quite well-know for the fact that there has been some criticism as to the domination of Esri within the GIS tools/resources within the geospatial science industry, but I believe it is something we could take as a pinch of salt. Not everything in Esri's massive line of commercial products are superior to other platforms but it is a starting point for any new geospatialists who wants to explore technologies there are not familiar with. 

All in all, this MOOC is heaven-sent. For me, I have been playing with the web apps and web maps for close to 4 years and I can attest to the fact that it covers all the basics. For the developer's bit, maybe not so much as going through it in a distinct step-by-step but it does stoke the curiosity as to how it works. The question is, how do we make it work. Now that's a mystery I am eager to solve. 

I'm going to put this on my ever-expanding to-do list and think JavaScript for another few more months of testing out this ArcGIS API for JavaScript implementation. Tell me if you wanna know how this actually works and I'll share what I find out when I do.

For those who had missed out on this cohort, fear not. This MOOC runs twice a year and the next cohort is going to be from Feb 17 to March 17 2021. The registration is already open, so don’t hold back and click the link below:

Do-It-Yourself Geo Apps

Do register for a public account before signing up or just click 'Register' at the MOOC's page and it's open the open to either sign in or 'Create a public account'. It was a blast and I'm sure, if you've never used any of the feature I've mentioned above, you'll be as wide-eyed as I was 3 years ago. :D

Till then, stay spatially mappy comrades!

P/S: If you complete all the assignments and quizzes, you'll get a certificate of completion from Esri. Which is pretty rad!

take care

Studying is hard enough when you are fighting the onset of hormonal hurricane of emotions while you're a teenager. Then you have you formative 5 to 6 years of collegiate years where you learn that your idealistic notions of the world is just plain snobbish and that you're one of the many that finds satisfaction in pushing your ideas into other people's head. Doing all that while studying, it was hard too.

Then comes the job hunting, climbing ladders to nowhere from one workplace to another...just drifting. That was the vacation off from studying and just wanting to find stability.

Then one day, you broke your heart at the same time you get an opportunity to a decent work somewhere. You ran. Then it was about understanding this whole new dimension of knowledge that you've fallen in love with. It was like your first love came back to find you and tell you he's back and that he won't leave you alone anymore. So, back to the books you fall into.

So yeah...studying is a rollercoaster of emotions. Learning is a lifelong journey. You're never too old to learn something new. And then phrase where you can't teach old dogs new tricks is just plain inaccurate. You can teach them new tricks and they can do it if they want to. It's the matter of either they want to do it or if their body can still keep up with them.

If you're an older student, I feel you. You feel left behind at times and frustrated because you feel like you can't catch up with things. If you're someone who's starting to work on research studies, it may be a harder state of mind. Because you're almost always on your own lamenting on you're comprehension of theories and questioning if you get whatever you absorbed right. So...it is hard. How do you know that you're alright and that you're not spinning down the road of doomed isolation? You don't. These are some of the glaring things I think are the universal problems among people who, with the courage of a lion, answered the call of hermitage and subject themselves to an occupation so feared that it is deemed over once high school ends; student:

Wardrobe reduction to daily uniform of hoodies/sweaters and jog-pants/sweatpants

It's totally ok. If you're wearing the same one everyday for the rest of the week, you still win. It's practical, it's easy and for hijabsters, the hoodie helps with instant full head cover while you run to receive those stuff you endlessly ordered online. There's nothing to be ashamed off.

💡I say: Shower 🚿. Every. Day. Twice a day is most ideal. Stay fresh and hygienic. If you're not going anywhere nor are you working out in those attire, then you're good. At least for a week. When you shower, you're giving your body a break from the exhausting task of slouching, the heat of your brain going overdrive and your own face breaking out from the stress. So yeah...shower. Wash your face. Brush your teeth. You're gonna be at risk of letting yourself go when you're left alone too comfortable in your own company. If you're an introvert, it feels like heaven but it doesn't mean you should be at the risk of losing out on the joys of prepping yourself to look good once in a while. Regularly make the effort to find decent clothes you'll use to go out on a Saturday window shopping and kick back to relax. On the rare occasion you get a break from your endless paper chasing, Zoom calls and writing, you can use your wardrobe clearing day to match few clothing and have a runway at home yourself or for a date with yourself to the Netflix movie you're gonna watch on your laptop. Who says you can't dress up to do online shopping? That's the only way you'll know that you do not need that new shade of Forencos lipstick you saw the other day. Duh.

Sudden weight gain/loss

I can't say that is unexpected especially if you're the kind who can go long without exercises and the type who munch while you work (which is a bad habit by the way). But sometimes you can't help if especially if you live on your own or have your own living space. Being on your own makes it kind of easy to get off-track about the norm of things; how much have you eaten today...is that your 5th coffee...should you be finishing that leftovers...etc. It is harder if you don't have your own kitchen and rely on packed food or processed ones.

💡I say: I can assure you, it is NOT unusual. This happens often. Each one of us have different eating habits. There isn't a flexible advise to cater this problem. But it helps to think that when you're setting up a study environment, you're setting up a living environment that enable you to be inspired enough for productivity. I make it a point to have food available but at a good distance and exercise attire/equipment nearby. Better yet, wear them. To curb that crazy cravings, I try my best to have my meal on-time; breakfast at 8 am, lunch at 12 pm or 1 pm etc. Eat what you love, but in moderation. Hydrate regularly. It's hard, I know. All the chips🍟 and chocolate chip cookies🍪 and sweetened 3-in-1 lattes☕...I just...well, just remember...in moderation and you're a human. Don't beat yourself up if you eat them anyway.

Back pain, knee pain, frozen shoulder...everything hurts

I have a hard time tearing away from what I am doing for the 20-minutes-work-5-minutes-rest rule but I try my best to maintain good posture. The bad habit of crouching and slouching at desk jobs is a universal problem. At the onset of the pandemic, for the first time in my life, my back felt strained. I panicked and started to seriously stretch myself and found that not only I am killing my back, I was not breathing properly when I am doing my work due to my posture.

💡I say: Don't break the bank trying to buy new furniture or tools because from experience, you end up not actually using them for the purpose you bought them in the first place. Eg. treadmill...you're more likely to hang your clothes or towels there...or super expensive yoga mats. Carpet works just fine too.

Standing desk is deemed to have little to no effects on spinal/muscle health by some studies but I went on ahead and made a make-shift standing desk. I found that I'm more alert when I work standing and breathe better. There was no contest...standing desk is a choice made. My standing table is an old dining table that I turned into my bed where I put my portable laptop desk. So, it also works as a barre that I used to stretch my legs for that mid-splits I've been working on.

These are some of the stuff that plague me daily and how I cope. How about you guys? How do you take care of yourself as someone who willingly serve yourself on a platter to tertiary educational torture to console your ever-thirsty curious mind?

In ArcGIS Pro, the Erase tool only comes with the Advanced license. There are other ways to go about removing parts of a polygon/line data layer like the Clip tool. But Union is that tool where it makes more sense by principle.

It works by marking overlapping parts of two different data layer with integers; 1, 2 and so forth. Those that do not overlap is universally -1. So, remove everything else that you want out of the picture by deleting output features that contain FID integer values of more than -1! Simple eh?

Check out the <3 minutes demo below!

P/S: Happy New Year peeps! ♥

Eyes on the Forest Sarawak | WWF-Malaysia

The ‘Eyes on the Forest’ Sarawak web application is developed with the joint effort of WWF-Malaysia and WWF-Japan. Emulating the successful ‘Eyes on the Forest’ Sumatra database, this web app intends to provide an undiluted information on the rich diversity of Sarawak’s unique wildlife, forests and land uses as well as increasing the transparency on the threats; deforestation, infrastructure developments and urbanization, as well as its drivers; concessions, agricultural activities and unsupervised resource harvesting.

Identifying where the threats, the drivers and the conservation targets are located will help in making a concrete point at the policy level and push for more boots on the ground to ensure that the protected areas stay protected. The baseline information provided are essential in the prompt intervention for biodiversity protection.

Data collected and consolidated from dispersed public sources and engineered for simple and direct visualization. Values are generated based on verified legislated information provided by stakeholders and state agencies to the Sarawak Conservation Programme (SCP) for WWF-Malaysia.

Developed and maintained by:

Azalea Kamellia

GIS Officer (SCP) | WWF- Malaysia

June 2018 - Present

📑 International Climate Initiative (IKI) Land Use Plan: Green Initiative in the Heart of Borneo (HoB) Report

Tool: ArcGIS Pro 2.9.3 Technique: Overlay analysis, visualization via remote sensing technique

These maps are developed to aid or supplement the Natural Capital Valuation (NatCap) initiative. As cited by WWF:

An essential element of the Natural Capital Project is developing tools that help decision makers protect biodiversity and ecosystem services.

One of the site included in this initiative by WWF-Malaysia is the Heart of Borneo (HoB). Specifically for this exercise, the visualization of policy and land use eventually become the data input utilized in the tool InVest that generates the models and maps for the economic values of ecosystem services within the landscape of interest.

The generation of the data mainly includes superficial remote sensing to assess the status of the land use in the respective concessions using Sentinel-2 satellite image with specific band combination to identify tree cover, particularly mangrove forest.