{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Planet API Python Client\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial is an introduction to [Planet](https://www.planet.com)'s Data and Orders API using the official [Python client](https://github.com/planetlabs/planet-client-python), the `planet` module. It shows you how to create bulk orders, use our tools, and deliver to the cloud.\n",
    "\n",
    "## Requirements\n",
    "\n",
    "This tutorial assumes familiarity with the [Python](https://python.org) programming language throughout. Python modules used in this tutorial are:\n",
    "* [IPython](https://ipython.org/) and [Jupyter](https://jupyter.org/)\n",
    "* [planet](https://github.com/planetlabs/planet-client-python)\n",
    "* [geojsonio](https://pypi.python.org/pypi/geojsonio)\n",
    "* [rasterio](https://rasterio.readthedocs.io/en/latest/index.html)\n",
    "* [asyncio](https://docs.python.org/3/library/asyncio.html)\n",
    "\n",
    "You should also have an account on the Planet Platform and retrieve your API key from your [account page](https://www.planet.com/account/).\n",
    "\n",
    "## Useful links \n",
    "* [Planet Client V2 Documentation](https://github.com/planetlabs/planet-client-python)\n",
    "* [Planet Data API reference](https://developers.planet.com/docs/apis/data/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial will cover the basic operations possible with the Python client, particularly those that interact with the Data API and Orders API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up\n",
    "\n",
    "In order to interact with the Planet API using the client, we need to import the necessary packages & define helper functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#general packages\n",
    "import os\n",
    "import json\n",
    "import glob\n",
    "import asyncio\n",
    "import requests\n",
    "import nest_asyncio \n",
    "import matplotlib.pyplot as plt\n",
    "from requests.auth import HTTPBasicAuth\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "#geospatial packages\n",
    "import rasterio\n",
    "import geopandas as gpd\n",
    "from shapely.geometry import shape\n",
    "from shapely.ops import unary_union\n",
    "\n",
    "\n",
    "#planet SDK\n",
    "from planet import Auth, reporting, Session, OrdersClient, order_request, data_filter\n",
    "\n",
    "\n",
    "# We will also create a small helper function to print out JSON with proper indentation.\n",
    "def indent(data):\n",
    "    print(json.dumps(data, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We next need to create a `client` object registered with our API key. The API key will be automatically read from the `PL_API_KEY` environment variable if it exists. If not, you can provide it below. You can also authenticate via the CLI using [`auth init`](https://planet-sdk-for-python-v2.readthedocs.io/en/latest/cli/cli-reference/?h=auth#auth:~:text=message%20and%20exit.-,auth,-%C2%B6), this will store your API key as an environment variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if your Planet API Key is not set as an environment variable, you can paste it below\n",
    "if 'PL_API_KEY' in os.environ:\n",
    "    API_KEY = os.environ['PL_API_KEY']\n",
    "else:\n",
    "    API_KEY = 'PASTE_API_KEY_HERE'\n",
    "    os.environ['PL_API_KEY'] = API_KEY\n",
    "\n",
    "client = Auth.from_key(API_KEY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Searching\n",
    "\n",
    "We can search for items that are interesting by using the `quick_search` member function. Searches, however, always require a proper request that includes a filter that selects the specific items to return as seach results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also read in a GeoJSON geometry into a variable so we can use it during testing. The geometry can only have one polygon to work with the data API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"lake_shasta.geojson\") as f:\n",
    "    geom_all = json.loads(f.read())['features'][0]['geometry']\n",
    "geom_large = {\n",
    "    \"type\": \"Polygon\",\n",
    "    \"coordinates\": [\n",
    "        [[-122.48691050204125,40.7049384269217],\n",
    "         [-122.11474863680688,40.7049384269217],\n",
    "         [-122.11474863680688,40.898294348395034],\n",
    "         [-122.48691050204125,40.898294348395034],\n",
    "         [-122.48691050204125,40.7049384269217]]]\n",
    "}\n",
    "print(geom_all)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The possible filters include `and_filter`, `date_range_filter`, `range_filter` and so on, mirroring the options supported by the Planet API. Additional filters are described [here](https://planet-sdk-for-python-v2.readthedocs.io/en/latest/python/sdk-guide/#filter:~:text=(main())-,Filter,-%C2%B6)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the filters we'll use to find our data\n",
    "\n",
    "item_types = [\"PSScene\"]\n",
    "\n",
    "#Geometry filter\n",
    "geom_filter = data_filter.geometry_filter(geom_large)\n",
    "\n",
    "#Date range filter\n",
    "date_range_filter = data_filter.date_range_filter(\n",
    "    \"acquired\", gt = datetime(month=11, day=1, year=2022),\n",
    "    lt = datetime(month=3, day=1, year=2023))\n",
    "#Cloud cover filter\n",
    "#cloud_cover_filter = data_filter.range_filter('clear_percent', gt = 80)\n",
    "\n",
    "#Combine all of the filters\n",
    "combined_filter = data_filter.and_filter([geom_filter, date_range_filter])#, cloud_cover_filter])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "combined_filter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the method of directly pinging the API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "item_type = \"PSScene\"\n",
    "\n",
    "# API request object\n",
    "search_request = {\n",
    "  \"item_types\": [item_type], \n",
    "  \"filter\": combined_filter\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fire off the POST request\n",
    "search_result = \\\n",
    "  requests.post(\n",
    "    'https://api.planet.com/data/v1/quick-search',\n",
    "    auth=HTTPBasicAuth(API_KEY, ''),\n",
    "    json=search_request)\n",
    "\n",
    "# extract image IDs only\n",
    "image_ids = [feature['id'] for feature in search_result.json()['features']]\n",
    "print(len(image_ids))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pagination links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "search_result.json()['_links']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's do it using the SDK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "async with Session() as sess:\n",
    "    cl = sess.client('data')\n",
    "    item_list = [i async for i in cl.search(search_filter=combined_filter, item_types=item_types,limit=500)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the number of items requested is more than 250, the client will automatically fetch more pages of results in order to get the exact number requested.\n",
    "\n",
    "Then we can save the output to be visualized as a geojson"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(item_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Cloud cover filter\n",
    "cloud_cover_filter = data_filter.range_filter('clear_percent', gt = 80)\n",
    "\n",
    "#Combine all of the filters\n",
    "combined_filter = data_filter.and_filter([geom_filter, date_range_filter, cloud_cover_filter])\n",
    "\n",
    "async with Session() as sess:\n",
    "    cl = sess.client('data')\n",
    "    item_list = [i async for i in cl.search(search_filter=combined_filter, item_types=item_types,limit=500)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(item_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can iterate through our search results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for item in item_list:\n",
    "    print(item['id'], item['properties']['item_type'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can save all of our scene footprints as a geojson"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scene_geoms = {\n",
    "  \"type\": \"FeatureCollection\",\n",
    "  \"features\": []\n",
    "}\n",
    "\n",
    "if not os.path.isdir('output'):\n",
    "    os.mkdir('output')\n",
    "else:\n",
    "    if os.path.isfile('output/results.geojson'):\n",
    "        os.remove('output/results.geojson')\n",
    "\n",
    "with open('output/results.geojson','w') as f:\n",
    "    for item in item_list:\n",
    "        geom_out =     {\n",
    "          \"type\": \"Feature\",\n",
    "          \"properties\": item['properties'],\n",
    "          \"geometry\": item['geometry']\n",
    "        }\n",
    "        scene_geoms['features'].append(geom_out)\n",
    "    jsonStr = json.dumps(scene_geoms)\n",
    "    f.write(jsonStr)\n",
    "    f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "item_list[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ordering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have all of the imagery that we want to order we need to package it in a way that the Orders API can handle. Breaking it up by week."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grouped_items = []\n",
    "current_group = []\n",
    "#reverse the list since it comes in last date first\n",
    "reversed_items = sorted(item_list, key=lambda item: item['properties']['acquired'])\n",
    "\n",
    "#Select the earliest item\n",
    "group_start_date = datetime.strptime(reversed_items[0]['properties']['acquired'], \"%Y-%m-%dT%H:%M:%S.%fZ\")\n",
    "\n",
    "for item in reversed_items:\n",
    "    time_object = item['properties']['acquired']\n",
    "    time = datetime.strptime(time_object, \"%Y-%m-%dT%H:%M:%S.%fZ\")\n",
    "    \n",
    "    if time < group_start_date + timedelta(days=7):\n",
    "        current_group.append(item)\n",
    "        \n",
    "    else:\n",
    "        grouped_items.append(current_group)\n",
    "        current_group = [item]\n",
    "        group_start_date = time\n",
    "if current_group:\n",
    "    grouped_items.append(current_group)\n",
    "\n",
    "print(len(grouped_items))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets see what the cloud cover is for our scenes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for item in grouped_items[0]:\n",
    "    print(item['properties']['clear_percent'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to sort the images to prioritize the clearer images ontop for the next step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_items = []\n",
    "for group in grouped_items:\n",
    "    sorted_group = sorted(group, key=lambda item: item['properties']['clear_percent'], reverse=True)\n",
    "    sorted_items.append(sorted_group)\n",
    "    \n",
    "\n",
    "for item in sorted_items[0]:\n",
    "    print(item['properties']['clear_percent'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_overlap(geometry1, geometry2):\n",
    "    \"\"\"Calculate the area of overlap between two geometries.\"\"\"\n",
    "    shape1 = shape(geometry1)\n",
    "    shape2 = shape(geometry2)\n",
    "\n",
    "    # Compute the intersection of the two geometries.\n",
    "    intersection = shape1.intersection(shape2)\n",
    "\n",
    "    return intersection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look for the minimum about on scenes to cover the entire AOI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "minimum_sorted_list = []\n",
    "\n",
    "\n",
    "for week_items in sorted_items:\n",
    "    intersection = False\n",
    "    weekly_minimum_list = []\n",
    "    for item in week_items:\n",
    "        #for each scene itterate through every geometry and check if it overlaps with the scene\n",
    "        overlap = get_overlap(geom_all, item['geometry'])\n",
    "        if intersection:\n",
    "            new_intersection = unary_union([overlap,intersection])\n",
    "\n",
    "            #If the new interseciton is bigger then the old then add the scene to the order\n",
    "            if round(new_intersection.area, 8) > round(intersection.area, 8):\n",
    "                intersection = new_intersection\n",
    "                weekly_minimum_list.append(item)\n",
    "        else:\n",
    "            if overlap.area > 0:\n",
    "                intersection = overlap\n",
    "                weekly_minimum_list.append(item)\n",
    "    print(len(week_items), \" to \", len(weekly_minimum_list))\n",
    "    \n",
    "    if len(weekly_minimum_list) > 0:\n",
    "        minimum_sorted_list.append(weekly_minimum_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for item in sorted_items[0]:\n",
    "    print(item['properties']['clear_percent'])\n",
    "print(\"Now\")\n",
    "for item in minimum_sorted_list[0]:\n",
    "    print(item['properties']['clear_percent'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets print the average clear percent of each scene we are ordering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for group in minimum_sorted_list:\n",
    "    clear = []\n",
    "    for item in group:\n",
    "        clear.append(int(item['properties']['clear_percent']))\n",
    "    print(sum(clear)/len(clear))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to reverse the order of the scenes one more time because when mosaicing the last scene is stacked at the top"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "order_items = []\n",
    "for group in minimum_sorted_list:\n",
    "    sorted_group = sorted(group, key=lambda item: item['properties']['clear_percent'])\n",
    "    order_items.append(sorted_group)\n",
    "    \n",
    "for item in order_items[0]:\n",
    "    print(item['properties']['clear_percent'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Place a Order\n",
    "Create the order structure using `planet` functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "async def assemble_order(name,item_ids):\n",
    "    products = [\n",
    "        order_request.product(item_ids, 'analytic_udm2', 'PSScene')\n",
    "    ]\n",
    "    \n",
    "    clip = order_request.clip_tool(aoi=geom_all)\n",
    "    bandmath = order_request.band_math_tool(b1='(b2-b4)/(b2+b4)*100+100', pixel_type='8U')\n",
    "    composite = order_request.composite_tool()\n",
    "\n",
    "    \n",
    "    \n",
    "    tools = [clip,bandmath,composite]\n",
    "\n",
    "    request = order_request.build_request(\n",
    "        name, products=products, tools=tools)\n",
    "    return request\n",
    "    \n",
    "request =  await assemble_order(\"test\",['20230207_180504_51_24b6'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(request)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets create a funcion to order imagery"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "async def do_order(request):\n",
    "    async with Session() as sess:\n",
    "        cl = OrdersClient(sess)\n",
    "        #with reporting.StateBar(state='creating') as bar:\n",
    "        order = await cl.create_order(request)\n",
    "        #bar.update(state='created', order_id=order['id'])\n",
    "\n",
    "        await cl.wait(order['id'],max_attempts=0)#, callback=bar.update_state)\n",
    "        os.mkdir(request['name'])\n",
    "        \n",
    "        # if we get here that means the order completed. Yay! Download the files.\n",
    "        await cl.download_order(order['id'],directory=request['name'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can create all our orders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "order_list = []\n",
    "folder_list= []\n",
    "name = \"lake_shasta_cloud_\"\n",
    "for group in order_items:\n",
    "    ids = []\n",
    "    order_name = name + group[0]['properties']['acquired'][:10]\n",
    "    print(order_name)\n",
    "    folder_list.append(order_name)\n",
    "    for item in group:\n",
    "        ids.append(item['id'])\n",
    "    order_list.append(await assemble_order(order_name,ids))\n",
    "print(len(order_list))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nest_asyncio.apply()\n",
    "\n",
    "#now all you need to do to have them run in parallel is to create an array of order requests\n",
    "async with Session() as sess:\n",
    "    tasks = [do_order(o) for o in order_list]\n",
    "    await asyncio.gather(*tasks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets visualize our output!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "files = []\n",
    "# for folder in folder_list:\n",
    "#     files.extend(glob.glob(folder+\"/*/composite.tif\"))\n",
    "\n",
    "files.extend(glob.glob(\"*/*/composite.tif\"))\n",
    "files.sort()\n",
    "nrow = 4\n",
    "ncol = 3\n",
    "\n",
    "f, axes = plt.subplots(nrow, ncol, figsize=(3*ncol, 3*nrow))\n",
    "for file, ax in zip(files, axes.flatten()):\n",
    "    with rasterio.open(file) as src:\n",
    "        arr = src.read()\n",
    "    \n",
    "    ax.imshow(arr[0], cmap=\"GnBu\")\n",
    "    \n",
    "    \n",
    "    date = file.split(\"_\")[-1].split('/')[0]\n",
    "    ax.set_title(date)\n",
    "\n",
    "for ax in axes.flatten():\n",
    "    ax.axis(\"off\")\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Google Earth Engine Delivery\n",
    "\n",
    "In order to deliver straight to earth engine you need to connect your gee account to a google cloud project and authorize planet to deliver to it. [Here](https://developers.planet.com/docs/integrations/gee/quickstart/) are some instrucitons to help with the process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "async def assemble_order(item_ids, delivery=None):\n",
    "    products = [\n",
    "        order_request.product(item_ids, 'analytic_8b_sr_udm2', 'PSScene')\n",
    "    ]\n",
    "    tools = [order_request.clip_tool(aoi=geom_all)]\n",
    "    request = order_request.build_request(\n",
    "        'test_order_sdk', products=products, tools=tools, delivery=delivery)\n",
    "    return request\n",
    "\n",
    "\n",
    "delivery = order_request.google_earth_engine(\n",
    "    project=\"planet-services-staging\",\n",
    "    collection=\"gis-day-gee-demo\")\n",
    "\n",
    "\n",
    "item_ids = [item[\"id\"] for item in item_list]\n",
    "request =  await assemble_order(item_ids, delivery=delivery)\n",
    "\n",
    "\n",
    "# an async Orders client to request order creation\n",
    "async with Session() as sess:\n",
    "    cl = OrdersClient(sess)\n",
    "    with reporting.StateBar(state='creating') as bar:\n",
    "        # create order via Orders client\n",
    "        order = await cl.create_order(request)\n",
    "        bar.update(state='created', order_id=order['id'])\n",
    "        await cl.wait(order['id'], callback=bar.update_state)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
