Using Wreq and Lens libraries to query Prosper for account info

By Posted on

Problem

I am trying to create a program to manage my Prosper investments for me as an intro to Haskell project. I created a post a few weeks ago for some of the simpler backend calculations: Recommend Next Note Purchase

Now I’ve started working on the web APIs in order to retrieve my account information from Prosper. Though it may not look it, the code below has already undergone some major refactoring to attempt to bring it to slightly readable.

This is my first attempt at any RESTful service interaction, JSON parsing, and usage of Lens. I have a few areas of this code I would really appreciate some feedback.

  1. I feel like I really went overboard with the let bindings, but I couldn’t figure out how to break up some absurdly long lines in another way.

  2. All my maybeXXX functions. I needed them to operate on the Maybe typeclass in the way I was expecting, but I feel like there is a more elegant use of Monads that could avoid these auxiliary functions.

  3. The creation of the MResult type was done just to parse the response body of the web API. There must be a way to use Lens to extract that field and parse it as a list of Note directly.

Any other feedback on the general structure or style of the code is greatly appreciated.

{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE DeriveGeneric #-}

import Prosper
import Network.Wreq
import Control.Lens
import qualified Data.ByteString as BS
import qualified Data.ByteString.Lazy.Internal as BL
import qualified Data.Text as T
import Data.Text.Encoding
import Data.Aeson
import Data.Aeson.Lens 
import GHC.Generics

data Note = Note {
    loan_number :: Int,
    prosper_rating :: T.Text
    } deriving (Generic, Show)
instance FromJSON Note

data MResult = MResult {
    result :: [Note]
    } deriving (Generic, Show)
instance FromJSON MResult

-- String Constants
prosperAddress = "https://api.prosper.com/v1/"
authTarget = "security/oauth/token"
accountTarget = "accounts/prosper/"
notesTarget = "notes/"
clientID = "cicicicici"
clientSecret = "ssssssssssssssss"
userID = "uuuuuuuuuu"
password = "pppppppppp"

body :: BS.ByteString
body =  strToBS $
            "grant_type=password" ++ 
            "&client_id=" ++ clientID ++ 
            "&client_secret=" ++ clientSecret ++ 
            "&username=" ++ userID ++ 
            "&password=" ++ password
    where strToBS = encodeUtf8 . T.pack

-- Retrives the OAuth2 token from the Prosper server
oauthToken :: IO T.Text
oauthToken = do
    let opts = defaults & header "Accept" .~ ["application/json"]
                        & header "Content-type" .~ ["application/x-www-    form-urlencoded"]
    resp <- postWith opts (prosperAddress ++ authTarget) body
    return $ resp ^. responseBody . key "access_token" . _String

-- Performs a GET request to the provided target using the provided OAuth     token
getTarget target token = do
    let tokenStr = encodeUtf8 $ T.append "bearer " token
    let opts = defaults & header "Authorization" .~ [tokenStr]
    getWith opts (prosperAddress ++ target)     

-- Retrives the list of currently owned notes from Prosper
-- Note: The query responses are paginated to 25 results per response
getNotesList :: T.Text -> Int -> IO [T.Text]
getNotesList token offset = do
    let target = notesTarget ++ "?offset=" ++ (show offset)
    resp <- getTarget target token
    let r =  fmap result (decode $ resp ^. responseBody :: Maybe MResult)
    let notes = fmap (map prosper_rating) r
    if (maybeLength notes) /= 25 then
        maybeReturn notes
        else do
            restOfNotes <- getNotesList token (offset+25)
            return $ maybeAppend notes restOfNotes

maybeLength Nothing = 0
maybeLength (Just xs) = length xs

maybeReturn Nothing = return []
maybeReturn (Just xs) = return xs

maybeAppend :: Maybe [a] -> [a] -> [a]
maybeAppend Nothing xs = xs
maybeAppend (Just ys) xs = ys ++ xs

main = do
    token <- oauthToken 
    notes <- getNotesList token 0
    print $ recommendNote notes [0,0,0.2,0.2,0.25,0.30,0.05]

Solution

You’re on the right path! You’re asking great questions.

I feel like I really went overboard with the let bindings, but I couldn’t figure out how to break up some absurdly long lines in another way.

This is sort of unavoidable with wreq (or any HTTP client library in general probably), as you have to pass around a lot of state to communicate on a stateless protocol. You do a little better if you pass your options as a single Options value. Leads to friendlier type signatures too.

All my maybeXXX functions. I needed them to operate on the Maybe typeclass in the way I was expecting, but I feel like there is a more elegant use of Monads that could avoid these auxiliary functions.

You can use fmap or <$> instead of constructing these maybeWobble helpers, but as we see below there’s an even better option.

The creation of the MResult type was done just to parse the response body of the web API. There must be a way to use Lens to extract that field and parse it as a list of Note directly.

There is! Your usage of the lens-aeson package is right on the money. To grab all the prosper_rating :: Text of all the notes in the result key, we can simply write this lens traversal:

_ratings = responseBody . key "result"
                        . _Array
                        . traverse
                        . key "prosper_rating"
                        . _String

(Traversal just means that you need to use ^.. instead of ^. and that you’ll get back a list of values instead of one when you do so.)

The other way you could be using more lens is to import the text lens module, which provides two “once you learn of them you can’t stop using them” helpers:

  • packed, unpacked: isomorphisms between Text and String. Never import qualified Data.Text again!

  • utf8: a prism that converts a ByteString to a Text, when possible. Never import qualified Data.Bytestring again!

  • _Show: a prism (in Control.Lens actually, but we’ll toss it onto this list) that lets you convert any Show a => a value into String. Earn a combo bonus when you compose it with packed to get a Text.

One other note is that you should use param and := from the wreq library to construct parameter strings and POST bodies. They handle the text massaging for you.

OK, putting everything together:

{-# LANGUAGE OverloadedStrings #-}

module Free_D where

import Control.Lens
import Data.Aeson
import Data.Aeson.Lens
import Data.Monoid ((<>))
import Data.Text.Strict.Lens
import Network.Wreq

import Data.Text (Text)

-- String Constants
prosperAddress, clientID, clientSecret, userID, password :: Text
prosperAddress = "https://api.prosper.com/v1/"
clientID = "cicicicici"
clientSecret = "ssssssssssssssss"
userID = "uuuuuuuuuu"
password = "pppppppppp"

prosperURL :: Text -> String
prosperURL target = (prosperAddress <> target) ^. unpacked

-- | Retrives the OAuth2 token from the Prosper server.
oauthToken :: IO Text
oauthToken = do
    let opts =
          defaults
            & header "Accept" .~ ["application/json"]
            & header "Content-Type" .~ ["application/x-www-form-urlencoded"]
    let body =
          [ "grant_type" := ("password" :: Text)
          , "client_id" := clientID
          , "client_secret" := clientSecret
          , "username" := userID
          , "password" := password]
    resp <- postWith opts (prosperURL "security/oauth/token") body
    return (resp ^. responseBody . key "access_token" . _String)

-- | Performs a GET request to the provided target using the provided
-- OAuth token.
getTarget :: Options -> String -> Text -> IO (Response Value)
getTarget opts url token = do
    getWith (opts & header "Authorization" .~ rawr) url >>= asJSON
    where
        rawr = ["bearer " <> utf8 # token]

-- | Retrives the list of currently owned notes from Prosper.
--
-- Note: The query responses are paginated to 25 results per response.
getNotesList :: Text -> Int -> IO [Text]
getNotesList token offset = do
    resp <- getTarget opts (prosperURL "notes/") token
    let ratings = resp ^.. _ratings
    remainder <- if length ratings == 25 then return [] else getNotesList token (offset + 25)
    return (ratings ++ remainder)
    where
        opts = defaults & param "offset" .~ [offset ^. re _Show . packed]
        _ratings = responseBody . key "result" . _Array . traverse . key "prosper_rating" . _String

main :: IO ()
main = do
    token <- oauthToken
    notes <- getNotesList token 0
    print notes

(I got the code compiling but, as I don’t have an account, I was unable to run and test it.)

Leave a Reply

Your email address will not be published. Required fields are marked *