Hi @maxim,
Thanks for the response, but I’m afraid I’m not sure what you mean and how this would help my scenario.
I mispoke, I don’t have any timeouts set on my Workflow, rather on my Activities. I let the activities retry up to 30 seconds before failing the activity.
private final static ActivityOptions options = ActivityOptions.newBuilder()
.setScheduleToCloseTimeout(Duration.ofSeconds(30))
.setRetryOptions(RetryOptions.newBuilder()
.setBackoffCoefficient(2)
.build())
.build();
I understand the part about not setting a Workflow timeout, but unclear how the “update” would help me in the case where my backend service dies before the workflow calls the compensation routine to reverse my gift card charge.
My workflow calls my backend service via an “GiftCardActivity” which makes a REST call to a microservice running on another machine.
I am thinking about running each microservice as a Activity worker on each machine since it sounds like that might be better for resiliency?
If I understand correctly, if the Workflow worker tries to run the GiftCardActivity and that service is down, it will be completed when the remote worker service comes back online due to Temporal putting the activity request in the Task Queue?
If so, are there examples of running Activities in their own Worker process and the Workflow in another machine? Or do they both have to be run together? In which case that means I need to globally share the Workflow implementation across projects?
I’m a little confused if I can run Activities on one machine and the Workflow implementation on another.
What I’d ultimately like to do (if possible) is:
- User calls rest end point to kick off workflow
- Workflow starts
- Calls
SupportSiteActivity - OK response
- Calls
BCFactoryActivity - OK response
- Calls
VerifyGiftCardModelsActivity - OK response
- Loops through each gift card by calling
GiftCardActivity - Error after first card is successful because backend service is dead
- Wait for 30 seconds for response from GiftCardActivity
- Timeout activity since it’s dead
Now, the part I’m not sure of is I want the workflow to either stay running and not mark it as failed? so it can try the giftcard rollback when the service comes up AND return a response back to the user.
I can’t seem to have a workflow keep running AND return a response back to the user.
Also, when I restart my giftcard service the workflow doesn’t try to run the compensations it couldn’t run previously due to the giftcard service being down.
I did manage to get my giftcard service running as a separate worker on another machine with a separate queue name, but I am still seeing the same issues as with the activity running locally.
Am I thinking about this the wrong way? it is sort of a sync/async paradigm where we want synchronous method calls to the services BUT also want the workflow to pick up where any service errors happened (if they do) AND be able to return a response to the user so we can notify them that the process failed and will continue cleanup when the services come back online.
Below is my Workflow which is kicked off from a REST call (Gateway):
Gateway
@Path("")
public class LicensingGateway {
@ConfigProperty(name = "temporal.purchase.license.task.queue")
String taskQueue;
@Inject
Logger log;
@Inject
WorkflowApplicationObserver observer;
@Path("/purchase")
@Timed
@POST
@AddingSpanAttributes
public Response purchaseLicenses(@Valid @SpanAttribute(value = "http.payload") PurchaseLicenseRequest request) {
log.infof("Attempting to purchase licenses with request [%s]", request);
UUID requestId = UUID.randomUUID();
PurchaseLicenseWorkflow workflow = observer.getClient().newWorkflowStub(
PurchaseLicenseWorkflow.class, WorkflowOptions.newBuilder()
.setWorkflowId("PurchaseLicenseRequest-" + requestId.toString())
.setTaskQueue(taskQueue).build()
);
PurchaseLicenseResponse output = workflow.purchaseLicenses(requestId, request);
return Response.ok(output).build();
}
}
WorkFlow
public class PurchaseLicenseWorkflowImpl implements PurchaseLicenseWorkflow {
private static final Logger log = Workflow.getLogger(PurchaseLicenseWorkflowImpl.class);
SupportSiteActivities supportSiteActivity = ActivityStubsProvider.getSupportSiteActivities();
GiftCardActivities giftCardActivity = ActivityStubsProvider.getGiftCardActivities();
LicenseServiceActivities licenseActivity = ActivityStubsProvider.getLicenseActivities();
BCFactoryActivities bcfactoryActivity = ActivityStubsProvider.getBCFactoryActivities();
VerificationActivities verificationActivity = ActivityStubsProvider.getVerificationActivities();
@Override
public PurchaseLicenseResponse purchaseLicenses(UUID requestId, PurchaseLicenseRequest request) {
Objects.requireNonNull(requestId, "A unique request id is required");
Objects.requireNonNull(request, "PurchaseLicenseRequest is required");
// Controls the rollbacks
Saga saga = new Saga(new Saga.Options.Builder().build());
try {
// verify user
// Will throw exception if user not found
log.debug("Calling Support Site to verify user {}", request.getOwnerEmail());
supportSiteActivity.verifyUser(request.getOwnerEmail());
// verify the serials exist
// and group them by model AG1->SERIALS, etc
// will throw exception if any serials not found
log.debug("Calling BCFactory to group serial numbers by model");
GroupSerialsByModelResponse groupedSerials = bcfactoryActivity.groupSerialsByModel(request.getSerials());
// Verify that we have the proper serial models to gift card types
// in the request
// will throw exception if any mismatches
log.debug("Verifying gift card and serial model matches");
verificationActivity.verifySerialModelsToGiftCardTypes(groupedSerials, request.getGiftCards());
// Debit the cards
// If one fails, they all fail and will
// be rolled back
log.debug("Calling Gift Card service to debit {} gift cards", request.getGiftCards().size());
debitGiftCards(request, saga, requestId);
// Generate the licenses
// if this fails, will cause a rollback of gift card transactions
log.debug("Calling license service to generate licenses");
Map<String, String> licenses = licenseActivity.generateLicenses(request).getLicenses();
// The response
PurchaseLicenseResponse lr = new PurchaseLicenseResponse();
lr.setOwnerEmail(request.getOwnerEmail());
lr.setExpirationDate(request.getExpirationDate());
lr.setRequestedSerials(request.getSerials());
lr.setLicenses(licenses);
log.debug("Process complete..Returning reponse - {}", lr);
return lr;
} catch (Exception e) {
saga.compensate();
throw ApplicationFailure.newFailureWithCause(e.getMessage(), "RUNTIME ERROR!", e);
}
}
/**
* Process the gift cards specified
*
* @param request
* @param saga
* @param requestId
*/
private void debitGiftCards(PurchaseLicenseRequest request, Saga saga, UUID requestId) {
// Charge each card
for (GiftCard giftCard : request.getGiftCards()) {
// Create new request
DebitAccountRequest r = new DebitAccountRequest();
r.setCardId(giftCard.getId());
r.setCredits(giftCard.getCredits());
r.setOwnerEmail(request.getOwnerEmail());
r.setInitiatedBy(request.getOwnerEmail());
r.setCardType(giftCard.getType());
// Perform operation
giftCardActivity.debitGiftCard(requestId, r);
// Setup the compensations in case of failures
saga.addCompensation(() -> giftCardActivity.reverseGiftCardTransaction(requestId, giftCard.getId()));
}
}